Fish, Chips and Ketchup

Fish, chips and the International Herald Tribune

Fish, chips and the International Herald Tribune

During his PhD years in Edinburgh, Geoffrey and his experimental psychology chums would often stop by the chippy after a night on the town. Geoffrey would queue up with his order of x1 pieces of fish, x2 lots of chips and x3 sachets of ketchup (yes, they charge for ketchup in Edinburgh!). Unable to focus his blurry eyes on the price list, he would estimate what the total would come to in order to ensure he had enough cash.

If he had been able to remember all the previous ordering history (‘first occasion: 3 pieces of fish, 4 lots of chips and 2 sachets of ketchup cost £1.10’), he would have been able solve the problem exactly after a few visits. But he didn’t – he just remembered the best guesses after the previous visit to the chippy.

But no worries. He treated the problem as a linear neural network and knew how to modify his best guesses after each visit well. He was also lucky in choosing a learning rate, ε, of 0.05 and so it only took 18 visits to the chippy before he was within tuppence of the right amount which he thought was good enough.

This almost certainly doesn’t bear any resemblance to the reality of why Prof Hinton (the ‘Godfather of Deep Learning’) chose to teach linear neural networks with an introductory example of fish, chips and ketchup.

But explaining how it works through a mathematical explanation of ‘the delta rule’ for fast ‘gradient descent’

∆wi= ε xi (t-y)

…is beyond most people whereas a large number of school child now learn to program in Python. I think playing around with some Python code would be a demystifying introduction to neural networks for many. So here is some code to help with this…

############################
# fish_chips_and_ketchup.py
############################
"""
A very simple example of the learning of a linear neural network
"""
# This is coded explicitly for fish, chips and ketchup
# for teaching clarity rather than being generalized.

from numpy  import exp      # For setting the learning rate
from random import randint  # For generating random chippy orders
MAX_ITERATIONS = 2000 # Number of visits to the chippy before giving up.
START_PRINTS   = 10   # Number of iterations reported on at the start.
STOP_ERROR     = 0.03 # Error margin - good enough to stop
cost = {'fish': 0.20, 'chips': 0.10, 'ketchup': 0.05} # This is the menu

def print_status_line(iteration, price, error): # Reporting of results at each iteration
    print ("%4d  Fish £%.2f, Chips £%.2f, Ketchup £%.2f, error £%.2f"
           % (iteration, price['fish'], price['chips'], price['ketchup'], error))

for e in range(1,7):
   # Set the learning rate 'epsilon' to exponentially slower values at each iteration
   epsilon = exp(-e)
   print ("Case %d: learning rate = %.3f" % (e, epsilon))

   weight = {'fish': 0.30, 'chips': 0.05, 'ketchup': 0.02} # Initial guesses
   error = (abs(weight['fish']-cost['fish'])
          + abs(weight['chips']-cost['chips'])
          + abs(weight['ketchup']-cost['ketchup']))
   print_status_line(0, weight, error)

   for n in range(1, MAX_ITERATIONS+1):
      # Just randomly set what this particular menu order is...
      portions = {'fish': randint(1, 5), 'chips': randint(1, 5), 'ketchup': randint(1, 5)}
      target_price = (weight['fish']*portions['fish']
                    + weight['chips']*portions['chips']
                    + weight['ketchup']*portions['ketchup'])
      actual_price = (portions['fish']*cost['fish']
                    + portions['chips']*cost['chips']
                    + portions['ketchup']*cost['ketchup'])
      # Difference in output...
      residual_error = target_price - actual_price
      # Condition for halting loop...
      prev_error = error
      error = (abs(weight['fish']-cost['fish'])
             + abs(weight['chips']-cost['chips'])
             + abs(weight['ketchup']-cost['ketchup']))
      # Adjust the weights
      for i in ['fish', 'chips', 'ketchup']:
         delta_weight = epsilon * portions[i] * residual_error
         weight[i] -= delta_weight

      # Output display and automatic halting on divergence or convergence...
      if abs(error) > 4.0*abs(prev_error):
          print_status_line(n, weight, error)
          print ("      Halting because diverging")
          break
      if (error <= STOP_ERROR) :
          print_status_line(n, weight, error)
          print ("      Halting because converged")
          break
      if (n <= START_PRINTS):
          print_status_line(n, weight, error)
      if (n == MAX_ITERATIONS) :
          print_status_line(n, weight, error)
          print ("      Halting but not yet converged")

Note: this Python code is written for clarity – for understanding by people not intimately familiar with the Python language – rather than for conciseness and efficiency. It is unapologetically ‘unpythonic’.

Running it produces the output…

Case 1: learning rate = 0.368
   0  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   1  Fish £0.29, Chips £0.03, Ketchup £0.01, error £0.18
   2  Fish £0.32, Chips £0.06, Ketchup £0.05, error £0.19
   3  Fish £-0.71, Chips £-0.14, Ketchup £-0.78, error £0.16
   4  Fish £12.15, Chips £12.72, Ketchup £15.30, error £1.98
      Halting because diverging
Case 2: learning rate = 0.135
   0  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   1  Fish £0.32, Chips £0.08, Ketchup £0.04, error £0.18
   2  Fish £0.28, Chips £0.05, Ketchup £-0.04, error £0.15
   3  Fish £0.24, Chips £0.03, Ketchup £-0.06, error £0.23
   4  Fish £0.36, Chips £0.60, Ketchup £0.51, error £0.22
   5  Fish £-1.41, Chips £-2.35, Ketchup £-1.26, error £1.12
      Halting because diverging
Case 3: learning rate = 0.050
   0  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   1  Fish £0.22, Chips £0.00, Ketchup £0.00, error £0.18
   2  Fish £0.29, Chips £0.17, Ketchup £0.17, error £0.16
   3  Fish £0.12, Chips £-0.04, Ketchup £0.13, error £0.28
   4  Fish £0.33, Chips £0.13, Ketchup £0.17, error £0.29
   5  Fish £0.22, Chips £0.02, Ketchup £0.10, error £0.29
   6  Fish £0.22, Chips £0.02, Ketchup £0.10, error £0.15
   7  Fish £0.20, Chips £0.01, Ketchup £0.07, error £0.15
   8  Fish £0.21, Chips £0.07, Ketchup £0.12, error £0.12
   9  Fish £0.18, Chips £0.05, Ketchup £0.04, error £0.11
  10  Fish £0.19, Chips £0.06, Ketchup £0.06, error £0.08
  18  Fish £0.21, Chips £0.11, Ketchup £0.04, error £0.02
      Halting because converged
Case 4: learning rate = 0.018
   0  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   1  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   2  Fish £0.29, Chips £0.04, Ketchup £0.01, error £0.18
   3  Fish £0.30, Chips £0.07, Ketchup £0.04, error £0.18
   4  Fish £0.26, Chips £0.06, Ketchup £0.03, error £0.14
   5  Fish £0.25, Chips £0.06, Ketchup £0.03, error £0.11
   6  Fish £0.25, Chips £0.06, Ketchup £0.03, error £0.11
   7  Fish £0.26, Chips £0.07, Ketchup £0.04, error £0.11
   8  Fish £0.26, Chips £0.08, Ketchup £0.04, error £0.10
   9  Fish £0.26, Chips £0.08, Ketchup £0.04, error £0.09
  10  Fish £0.26, Chips £0.08, Ketchup £0.04, error £0.09
  44  Fish £0.22, Chips £0.09, Ketchup £0.05, error £0.03
      Halting because converged
Case 5: learning rate = 0.007
   0  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   1  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   2  Fish £0.30, Chips £0.06, Ketchup £0.02, error £0.18
   3  Fish £0.30, Chips £0.06, Ketchup £0.03, error £0.17
   4  Fish £0.30, Chips £0.06, Ketchup £0.02, error £0.17
   5  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.17
   6  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.17
   7  Fish £0.29, Chips £0.05, Ketchup £0.02, error £0.18
   8  Fish £0.29, Chips £0.05, Ketchup £0.02, error £0.18
   9  Fish £0.29, Chips £0.04, Ketchup £0.02, error £0.18
  10  Fish £0.29, Chips £0.04, Ketchup £0.01, error £0.18
 152  Fish £0.21, Chips £0.09, Ketchup £0.04, error £0.03
      Halting because converged
Case 6: learning rate = 0.002
   0  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   1  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   2  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   3  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   4  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   5  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   6  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   7  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   8  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
   9  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
  10  Fish £0.30, Chips £0.05, Ketchup £0.02, error £0.18
 389  Fish £0.21, Chips £0.09, Ketchup £0.04, error £0.03
      Halting because converged
http://www.cs.toronto.edu/~hinton

Hinton and python

… …which shows:

  1. How the error (the total mismatch between ‘Geoffrey’s’ best guesses and the actual costs) generally (but not always) decrease, leading towards the correct answer,
  2. Fewer iterations are required for faster learning rate (higher values of ε) but that the guesses actually diverge when ε increases beyond some particular point.

Incidently, Prof Hinton was also introduced to python at an early age…

Advertisements
This entry was posted in Uncategorized and tagged , , , . Bookmark the permalink.

One Response to Fish, Chips and Ketchup

  1. Pingback: Backpropagation | Headbirths

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s