WEBVTT 00:00:08.333 --> 00:00:09.033 Hello and 00:00:09.033 --> 00:00:12.183 welcome back to the final C2HO workshop, 00:00:12.183 --> 00:00:15.483 focusing on an introduction to machine learning for 00:00:15.483 --> 00:00:18.816 creative computing with ML5. 00:00:18.900 --> 00:00:21.533 The Ml5 library allows us to utilize machine 00:00:21.533 --> 00:00:25.666 learning directly in our web browser, using P5.js 00:00:25.733 --> 00:00:30.533 and the ML5 library also gives us access to a bunch of pre-trained models, and 00:00:30.533 --> 00:00:33.816 today it will be playing around with pose estimation 00:00:33.816 --> 00:00:36.816 using the hand pose model. 00:00:36.933 --> 00:00:42.333 Hand pose is a technology used to detect and track hand positions and gestures. 00:00:42.416 --> 00:00:47.766 It has a wide variety of applications from VR and AR experiences 00:00:47.850 --> 00:00:51.633 to human computer interaction and using the ML5 library, 00:00:51.633 --> 00:00:55.483 we can load a Pre-trained hand post model into our P5 script 00:00:55.566 --> 00:00:59.416 to detect and track or hand using the webcam. 00:00:59.483 --> 00:01:02.516 So let's begin by setting up our P5 sketch, 00:01:02.516 --> 00:01:06.333 which gives us access to the ML5 library. 00:01:06.416 --> 00:01:09.150 So in our index HTML file, 00:01:09.150 --> 00:01:13.500 we have loaded our ML5 library 00:01:13.583 --> 00:01:19.233 and since this is the fourth session of these two actual workshops, 00:01:19.316 --> 00:01:23.133 and while it's not necessary to have joined in the other sessions, 00:01:23.133 --> 00:01:27.850 I might not be covering all elements in equal detail each time. 00:01:27.933 --> 00:01:31.350 So if you find something confusing, you might find help. 00:01:31.350 --> 00:01:35.583 In a previous session. 00:01:35.666 --> 00:01:39.033 Let's start by setting up our canvas. 00:01:39.116 --> 00:01:42.300 So since we are tracking 00:01:42.300 --> 00:01:46.950 hand poses using our webcams, we'll also be loading up the video element. 00:01:46.950 --> 00:01:53.050 So make sure to allow your browser to access the webcam. 00:01:53.133 --> 00:01:55.083 So first I'm going to set up 00:01:55.083 --> 00:02:00.883 some canvas variables describing the width and height of the canvas. 00:02:00.900 --> 00:02:06.300 Since different screen sizes will work better with different canvas sizes. 00:02:06.300 --> 00:02:11.750 So please feel free to play around with whatever suits your setup. 00:02:11.833 --> 00:02:15.433 So like canvas width 00:02:15.516 --> 00:02:21.250 and canvas height 00:02:21.333 --> 00:02:24.333 and then we are going to use these variables 00:02:24.400 --> 00:02:31.250 instead inside our create canvas function 00:02:31.333 --> 00:02:33.933 like this, and then we're just going to declare them, 00:02:33.933 --> 00:02:37.550 give them some values inside our setup function. 00:02:37.633 --> 00:02:40.300 So canvas width. 00:02:40.300 --> 00:02:48.700 First we'll start with the default, 00:02:48.783 --> 00:02:51.433 something like this. 00:02:51.433 --> 00:02:55.850 And then let's set up our video element. 00:02:55.933 --> 00:02:59.750 So video 00:02:59.833 --> 00:03:05.416 equals create capture, 00:03:05.500 --> 00:03:08.083 which is a P5 function, 00:03:08.083 --> 00:03:10.933 and it requires 00:03:10.933 --> 00:03:14.283 the capitalized word VIDEO 00:03:14.383 --> 00:03:19.283 to refer to the default chosen video source. 00:03:19.350 --> 00:03:22.900 You can check out the P5 documentation if you want to use a different camera 00:03:22.900 --> 00:03:27.083 than your default webcam, things like that. 00:03:27.150 --> 00:03:29.533 And then we will set the size 00:03:29.533 --> 00:03:32.900 of this video element 00:03:32.966 --> 00:03:41.400 to the same size as our cameras. 00:03:41.483 --> 00:03:43.366 And the final thing we will do for 00:03:43.366 --> 00:03:47.466 now is to declare our hand post model. 00:03:47.550 --> 00:03:52.683 So we will call it hand pose this 00:03:52.766 --> 00:03:57.266 and we'll be calling that five hand 00:03:57.350 --> 00:04:00.816 pose model. 00:04:00.900 --> 00:04:03.316 We need to 00:04:03.316 --> 00:04:06.166 supply a video element 00:04:06.166 --> 00:04:09.166 for it to get its input from, 00:04:09.350 --> 00:04:13.416 and we will define a callback function 00:04:13.500 --> 00:04:16.900 model ready to deal with the 00:04:16.966 --> 00:04:23.066 the status of the model and let us know that the model has completed its loading. 00:04:23.150 --> 00:04:26.516 So let's just define model ready immediately. 00:04:26.516 --> 00:04:29.483 Function model 00:04:29.483 --> 00:04:36.266 ready. 00:04:36.350 --> 00:04:41.200 And for now let's just log 00:04:41.266 --> 00:04:47.500 that the model has loaded. 00:04:47.566 --> 00:04:48.650 So console 00:04:48.650 --> 00:04:52.300 log means that this message will show up down here 00:04:52.366 --> 00:04:55.366 in our little console. 00:04:55.500 --> 00:04:57.933 So if we now run this script, 00:04:57.933 --> 00:05:02.733 we can see that the model takes some time to load compared 00:05:02.783 --> 00:05:07.816 to the other pre-trained models that we've used so far in these workshops, 00:05:07.883 --> 00:05:10.766 Hand pose is a bit on the larger side 00:05:10.766 --> 00:05:14.083 and requires a little more memory. 00:05:14.166 --> 00:05:18.116 So you see now it has loaded and we're ready to go. 00:05:18.116 --> 00:05:19.116 So it's a bit slower 00:05:19.116 --> 00:05:24.000 than the other pre-trained models that are available through ML5. 00:05:24.083 --> 00:05:26.566 So let's start by 00:05:26.566 --> 00:05:28.550 getting the hand pose model to 00:05:28.550 --> 00:05:32.066 just start predicting based on our video inputs. 00:05:32.150 --> 00:05:38.933 So first let's hide the video feedback and just display the image onto our canvas 00:05:38.933 --> 00:05:43.016 so that we can draw key points on top of our video. 00:05:43.100 --> 00:05:44.750 So we'll do that by 00:05:44.750 --> 00:05:49.350 first adding a video height, 00:05:49.433 --> 00:05:56.966 video dot height, 00:05:57.050 --> 00:06:01.933 and then we are going to replace this call to background. 00:06:01.933 --> 00:06:05.966 And instead we are going to, in our draw function, 00:06:05.966 --> 00:06:10.083 we are going to call image which is a P5 function. 00:06:10.166 --> 00:06:13.166 We are going to supply it with our video. 00:06:13.333 --> 00:06:16.033 We are going to place the video element 00:06:16.033 --> 00:06:21.666 in the location given by X and Y here, zero zero. 00:06:21.733 --> 00:06:25.500 So that's the top left of our canvas 00:06:25.583 --> 00:06:28.983 and we need to give it our width and height 00:06:29.066 --> 00:06:32.900 so we can use the canvas width and canvas height. 00:06:32.900 --> 00:06:39.333 I'm just going to copy it from up here 00:06:39.416 --> 00:06:43.483 to avoid any spelling mistakes. 00:06:43.550 --> 00:06:45.383 So if we now press play, 00:06:45.383 --> 00:06:51.433 we should just have a single canvas which displays our video. 00:06:51.500 --> 00:06:55.633 There we go, I see I'm a little squished, 00:06:55.700 --> 00:06:59.433 so let's change the width 00:06:59.666 --> 00:07:02.666 to a little wider. 00:07:02.850 --> 00:07:08.583 Say say 480. 00:07:08.666 --> 00:07:11.000 Let's try that. 00:07:11.000 --> 00:07:13.983 That seems more reasonable. 00:07:13.983 --> 00:07:18.300 Again, change these values as you see fit. 00:07:18.366 --> 00:07:21.316 So to display our predictions, we'll use the hand 00:07:21.316 --> 00:07:25.316 pose on predict function from ML5. 00:07:25.400 --> 00:07:31.400 So the way we do that is after defining hand pose, 00:07:31.483 --> 00:07:38.933 we will right hand pose dot on 00:07:39.016 --> 00:07:42.300 and then we will give it 00:07:42.366 --> 00:07:43.333 the keyword. 00:07:43.333 --> 00:07:50.566 Predict 00:07:50.650 --> 00:07:55.950 and that will return some results. 00:07:56.016 --> 00:08:00.816 And so this is sort of a shorthand way of defining a function 00:08:00.816 --> 00:08:03.816 and how we want to deal with these results. 00:08:04.033 --> 00:08:07.700 So we put this small arrow 00:08:07.783 --> 00:08:10.333 and then we will define 00:08:10.333 --> 00:08:13.816 a list of predictions 00:08:13.900 --> 00:08:18.400 globally in our program to store up the predictions from the models. 00:08:18.400 --> 00:08:23.333 So let's define that here globally. 00:08:23.416 --> 00:08:27.950 So let predictions 00:08:28.033 --> 00:08:34.216 and just make that an empty list. 00:08:34.300 --> 00:08:38.483 And so now we can fill in this on projects 00:08:38.566 --> 00:08:43.050 we want to 00:08:43.116 --> 00:08:46.116 fill our predictions 00:08:46.333 --> 00:08:51.783 with whatever results we have received. 00:08:51.866 --> 00:08:53.550 Okay, so let's 00:08:53.550 --> 00:08:57.466 close off this call to predict. 00:08:57.533 --> 00:09:00.883 And I see that we have two different 00:09:00.966 --> 00:09:04.233 spellings for hand polls and since P5 is case 00:09:04.233 --> 00:09:07.233 sensitive, we should adjust that and make them the same. 00:09:07.433 --> 00:09:09.566 I also had to adjust my camera a little bit 00:09:09.566 --> 00:09:16.583 because we're using two cameras here and so now it should be hopefully fine. 00:09:16.666 --> 00:09:20.433 So let's run this. 00:09:20.516 --> 00:09:23.300 We get our camera up 00:09:23.300 --> 00:09:29.366 and yeah, nothing is really happening, 00:09:29.450 --> 00:09:30.916 but that's good. 00:09:30.916 --> 00:09:32.183 So there's no errors. 00:09:32.183 --> 00:09:37.250 And now we can use this list of predictions to draw the key points 00:09:37.250 --> 00:09:38.366 onto the canvas. 00:09:38.366 --> 00:09:43.050 So let's make a function called draw key points. 00:09:43.133 --> 00:09:48.033 So let's do that here. 00:09:48.116 --> 00:09:48.433 And I'm 00:09:48.433 --> 00:09:52.683 going to stop this video just because two videos continuously 00:09:52.700 --> 00:09:56.366 can be a bit annoying to watch. 00:09:56.450 --> 00:09:57.083 Let's see. 00:09:57.083 --> 00:10:00.083 Function draw 00:10:00.283 --> 00:10:17.033 key points. 00:10:17.116 --> 00:10:21.566 Now, if we look at the values that are returned by hand pose, 00:10:21.650 --> 00:10:25.683 we will see that they are in the form of a list of attributes. 00:10:25.766 --> 00:10:27.433 So let's have a look at that. 00:10:27.433 --> 00:10:31.066 By here in our results function, 00:10:31.066 --> 00:10:36.866 we can say console dot log 00:10:36.950 --> 00:10:47.516 predictions 00:10:47.600 --> 00:10:51.183 and then we can see down here in our console 00:10:51.266 --> 00:10:54.933 what the predictions actually look like. 00:10:55.016 --> 00:10:58.100 So before we continue drawing our key points, 00:10:58.250 --> 00:11:03.533 let's just have a look at how we will be handling these. 00:11:03.616 --> 00:11:06.616 So let's see 00:11:06.833 --> 00:11:08.083 if I hold my hand up. 00:11:08.083 --> 00:11:11.383 We see that there are some things happening here. 00:11:11.383 --> 00:11:14.383 We are receiving some results, some objects. 00:11:14.516 --> 00:11:17.816 So let's stop this and 00:11:17.900 --> 00:11:20.900 have a look at one of these objects. 00:11:21.016 --> 00:11:24.016 So within this 00:11:24.050 --> 00:11:27.600 main result, we get an object 00:11:27.683 --> 00:11:31.200 which has a bunch of attributes. 00:11:31.283 --> 00:11:33.383 So we get a confidence score. 00:11:33.383 --> 00:11:38.333 So how sure the model is that It is detecting a hand at all. 00:11:38.416 --> 00:11:41.650 There is a bounding box and some parameters. 00:11:41.666 --> 00:11:44.866 And then what we are really interested is this one, 00:11:44.866 --> 00:11:47.866 which is landmarks. 00:11:47.983 --> 00:11:51.050 So landmarks is an array 00:11:51.050 --> 00:11:55.466 which has 21 different elements in it. 00:11:55.550 --> 00:11:59.850 That's 21 key points on our hand. 00:11:59.933 --> 00:12:04.700 And what we can see here, if we look deeper into our our landmarks, 00:12:04.700 --> 00:12:08.433 we can see that each landmark consists of three values. 00:12:08.516 --> 00:12:12.983 So these are the X, Y, and Z positions of that key point. 00:12:13.066 --> 00:12:15.350 So what we'll be doing in our draw 00:12:15.350 --> 00:12:19.883 key points function is that we will be doing a nested 00:12:19.966 --> 00:12:22.950 for loop, which means that we will first 00:12:22.950 --> 00:12:26.133 traverse the predictions or our list of predictions, 00:12:26.216 --> 00:12:31.616 and then we will go through each key point detected within each prediction. 00:12:31.633 --> 00:12:35.516 So for each key points, we can then draw a small circle 00:12:35.516 --> 00:12:42.316 so that we can see exactly where it is tracking our hand. 00:12:42.383 --> 00:12:44.700 So let's start by 00:12:44.700 --> 00:12:49.716 removing this console.log so we don't have a bunch of text 00:12:49.800 --> 00:12:53.800 going on in our console 00:12:53.883 --> 00:12:58.983 and we can start filling in this draw key points function. 00:12:58.983 --> 00:13:01.983 So again, we're going to do a nested for loop. 00:13:02.066 --> 00:13:07.066 So for the outer, for loop, which is going to go through predictions, 00:13:07.133 --> 00:13:11.083 I'm going to say let I equals 00:13:11.150 --> 00:13:13.883 zero to begin with. 00:13:13.883 --> 00:13:17.700 And while I is less than the length of our predictions, 00:13:17.700 --> 00:13:21.216 so predictions 00:13:21.300 --> 00:13:25.416 Dot length, 00:13:25.500 --> 00:13:29.950 we will do whatever we want in the for loop 00:13:30.033 --> 00:13:33.466 and then we will do that until 00:13:33.533 --> 00:13:35.366 we reach a 00:13:35.366 --> 00:13:39.616 I equal to the length of our predictions list. 00:13:39.683 --> 00:13:44.500 And so we will increment I by one 00:13:44.583 --> 00:13:49.683 for each loop 00:13:49.766 --> 00:13:50.966 plus equals one. 00:13:50.966 --> 00:13:53.366 There we go. 00:13:53.366 --> 00:13:55.900 So this is our outer for loop. 00:13:55.983 --> 00:13:58.916 And so here let's 00:13:58.916 --> 00:14:01.683 get a hold of the current prediction 00:14:01.683 --> 00:14:05.283 in our list. 00:14:05.366 --> 00:14:09.916 So prediction equals predictions. 00:14:09.983 --> 00:14:18.516 So our list at the current spot. 00:14:18.600 --> 00:14:19.866 So for the current value of 00:14:19.866 --> 00:14:23.983 I and then we start our inner for loop. 00:14:24.066 --> 00:14:26.616 So that's for 00:14:26.616 --> 00:14:32.566 let's use a different variable name. So J 00:14:32.633 --> 00:14:36.016 And while J is 00:14:36.083 --> 00:14:40.600 less than 00:14:40.683 --> 00:14:44.383 predictions dot 00:14:44.466 --> 00:14:45.866 sorry, let's use the correct one. 00:14:45.866 --> 00:14:50.550 So this current prediction has some landmarks. 00:14:50.700 --> 00:14:58.566 So we want to go through each of these landmarks 00:14:58.650 --> 00:15:01.200 and we need to get the length 00:15:01.200 --> 00:15:02.983 of this one. 00:15:02.983 --> 00:15:05.616 Then we will 00:15:05.616 --> 00:15:07.833 increment that by one as well 00:15:07.833 --> 00:15:12.666 for each round. 00:15:12.750 --> 00:15:14.300 so now we are inside 00:15:14.300 --> 00:15:18.966 our nested for loop and 00:15:19.050 --> 00:15:20.316 just fix this. 00:15:20.316 --> 00:15:26.616 So we need to say let J, give it an initial value 00:15:26.700 --> 00:15:31.716 of zero to begin with. 00:15:31.800 --> 00:15:32.350 There we go. 00:15:32.350 --> 00:15:39.700 Now we are in it for loop and we can access this current key point. 00:15:39.766 --> 00:15:45.450 So let's say key point equals 00:15:45.533 --> 00:15:51.150 prediction dot landmark 00:15:51.233 --> 00:15:53.150 and give it that index 00:15:53.150 --> 00:16:01.166 that we are currently at. 00:16:01.250 --> 00:16:04.500 And now we can use the key point 00:16:04.500 --> 00:16:08.750 value at zero and at one, which is our X and Y. 00:16:08.750 --> 00:16:12.300 We won't really bother with the Z position 00:16:12.416 --> 00:16:16.366 for the session since we're just working in 2D anyway. 00:16:16.383 --> 00:16:17.866 But you can. 00:16:17.866 --> 00:16:21.483 So let's create 00:16:21.566 --> 00:16:23.116 a small circle 00:16:23.116 --> 00:16:26.883 using these x and y positions for each landmark. 00:16:26.966 --> 00:16:30.600 So we'll have to fill our little circle 00:16:30.600 --> 00:16:34.650 shapes with some color 00:16:34.733 --> 00:16:38.666 and let's just make them very green. 00:16:38.750 --> 00:16:41.300 Seems to be the standard for this. 00:16:41.300 --> 00:16:46.166 And we don't want any outline around them. 00:16:46.166 --> 00:16:49.500 So we're going to do no stroke 00:16:49.583 --> 00:16:50.600 and then 00:16:50.600 --> 00:16:53.816 we will use the Ellipse function 00:16:53.900 --> 00:16:56.483 from P5 to make a circle. 00:16:56.483 --> 00:17:01.066 And it requires us to say something 00:17:01.066 --> 00:17:05.766 about the X and Y position of the center of the Ellipse, 00:17:05.850 --> 00:17:08.633 and we need to give it a a width and height. 00:17:08.633 --> 00:17:12.116 So our exposition is in 00:17:12.166 --> 00:17:17.733 key point zero 00:17:17.816 --> 00:17:20.116 and our y position is, copy 00:17:20.116 --> 00:17:26.500 this, is in key point one. 00:17:26.566 --> 00:17:29.300 So it changes from 0 to 1 00:17:29.300 --> 00:17:33.300 and let's make it a perfect, perfect circle 00:17:33.383 --> 00:17:35.333 ten pixels 00:17:35.333 --> 00:17:39.150 each direction. 00:17:39.233 --> 00:17:43.950 And now we can add our call to draw key points 00:17:44.033 --> 00:17:47.616 inside our draw function. 00:17:47.683 --> 00:17:51.666 To draw key points. 00:17:51.733 --> 00:17:56.016 So we'll call this at every frame. 00:17:56.083 --> 00:17:59.583 And so let's save our project 00:17:59.666 --> 00:18:03.000 and then we can run it and have a look. 00:18:03.066 --> 00:18:06.616 So when I press play, 00:18:06.700 --> 00:18:08.900 I get my 00:18:08.900 --> 00:18:16.216 model loaded 00:18:16.300 --> 00:18:21.000 and let's see, 00:18:21.066 --> 00:18:24.066 we get an issue here. 00:18:24.283 --> 00:18:29.700 Let's check our console 00:18:29.766 --> 00:18:31.333 line 30. 00:18:31.333 --> 00:18:36.450 So let's see what that is. Yep. 00:18:36.516 --> 00:18:40.500 We need to refer to land marks, not landmark. 00:18:40.566 --> 00:18:47.450 Let's try again. 00:18:47.533 --> 00:18:50.933 Model is loaded, so we should be able to track our hand. 00:18:51.016 --> 00:18:53.833 Yeah. And it's tracking, 00:18:53.833 --> 00:18:58.366 but it's a bit skewed to one side. 00:18:58.366 --> 00:19:02.900 And from my experience, this happens if we are 00:19:02.983 --> 00:19:05.916 using a canvas 00:19:05.916 --> 00:19:10.400 dimension, which is slightly off. 00:19:10.483 --> 00:19:17.716 So to get it into that like 16:9 format, 00:19:17.800 --> 00:19:20.800 I usually just do 00:19:20.983 --> 00:19:26.100 640 x 480 00:19:26.166 --> 00:19:33.600 and that should fix the problem. 00:19:33.666 --> 00:19:34.566 Model is loaded 00:19:34.566 --> 00:19:40.400 and now it's tracking a lot better. 00:19:40.483 --> 00:19:42.950 So let's reset 00:19:42.950 --> 00:19:47.916 and make this look a little more interesting. 00:19:48.000 --> 00:19:50.033 So one of the things that we can do 00:19:50.033 --> 00:19:53.250 is to hide our video 00:19:53.250 --> 00:19:57.033 and focus just on the points that we are detecting. 00:19:57.116 --> 00:20:00.566 So let's implement two different visuals. 00:20:00.566 --> 00:20:03.566 One, to represent cold hands and one 00:20:03.566 --> 00:20:06.566 to represent hot hands, for example. 00:20:06.750 --> 00:20:12.700 So first, let's create some color gradients for us to work with. 00:20:12.766 --> 00:20:18.300 So we'll do that in our set up function 00:20:18.383 --> 00:20:23.850 so we can use what's called linear 00:20:23.933 --> 00:20:26.183 interpolation, 00:20:26.183 --> 00:20:28.583 often referred to as loop. 00:20:28.583 --> 00:20:33.116 So the way that we do that is we can 00:20:33.200 --> 00:20:35.300 set up two different 00:20:35.300 --> 00:20:37.850 numerical values and look between them. 00:20:37.850 --> 00:20:41.150 So to sort of move smoothly between two values. 00:20:41.400 --> 00:20:43.816 And we can also do that for our colors. 00:20:43.816 --> 00:20:46.016 So we create a sort of gradient. 00:20:46.016 --> 00:20:49.716 So first let's define our 00:20:49.800 --> 00:20:52.816 two sides of the of the spectrum here. 00:20:52.816 --> 00:20:55.733 So let's do this both for hot and cold colors. 00:20:55.733 --> 00:21:01.300 So let's say hot color from. 00:21:01.366 --> 00:21:04.800 And we're going to use the P5 00:21:04.883 --> 00:21:06.900 color function, 00:21:06.900 --> 00:21:11.666 which will give us a P5 color element and we need to give it some RGB values. 00:21:11.666 --> 00:21:15.450 So let's give it maximum of red 00:21:15.533 --> 00:21:19.416 and zero of green and zero of blue 00:21:19.500 --> 00:21:21.800 and then hot 00:21:21.800 --> 00:21:25.950 color to 00:21:26.033 --> 00:21:28.800 we make a second color, which should be on the other 00:21:28.800 --> 00:21:31.800 end of our interpolation scale 00:21:31.916 --> 00:21:34.950 and let's make that yellow. 00:21:34.966 --> 00:21:40.683 So that we need a maximum of red and a maximum of green and no blue. 00:21:40.766 --> 00:21:44.183 So that's our hot color from an hot color to. 00:21:44.183 --> 00:21:47.366 And we can do the same cold colors, so called 00:21:47.550 --> 00:21:51.366 color from. 00:21:51.450 --> 00:21:54.783 And then we can use 00:21:54.866 --> 00:21:57.800 just straight 00:21:57.800 --> 00:21:58.616 blue. 00:21:58.616 --> 00:22:01.733 So everything else set to zero red and green set to zero 00:22:01.733 --> 00:22:04.733 and just a high maximum blue. 00:22:04.766 --> 00:22:06.900 And then color. 00:22:06.900 --> 00:22:07.316 Let's see. 00:22:07.316 --> 00:22:11.283 Sorry, that should be 00:22:11.366 --> 00:22:16.466 called color to. 00:22:16.550 --> 00:22:20.133 And let's make that, 00:22:20.216 --> 00:22:23.583 for example, just a bright white. 00:22:23.666 --> 00:22:27.350 So we do that by setting our energy and be red, 00:22:27.350 --> 00:22:32.683 green and blue to their highest intensities. 00:22:32.766 --> 00:22:36.050 So now we can change the name of our draw. 00:22:36.066 --> 00:22:42.483 Key points function to specify that we are drawing cold hands. 00:22:42.483 --> 00:22:46.616 For example, start with the cold version. 00:22:46.683 --> 00:22:49.150 So draw cold 00:22:49.150 --> 00:22:51.366 hands. 00:22:51.366 --> 00:22:55.550 And in this fill function 00:22:55.633 --> 00:22:58.600 we are going to instead of sending in these RGB 00:22:58.600 --> 00:23:02.483 values directly, we're going to calculate a linear interpolation 00:23:02.566 --> 00:23:06.350 between the two maximum points. 00:23:06.433 --> 00:23:10.583 So the way we do that is we can define our color C 00:23:10.666 --> 00:23:16.400 as a perp color 00:23:16.483 --> 00:23:20.533 and lear color takes three arguments. 00:23:20.716 --> 00:23:26.000 So we need to give it our cold color 00:23:26.083 --> 00:23:30.083 00:23:30.166 --> 00:23:35.033 Our cold color from 00:23:35.116 --> 00:23:40.333 and are cold color to. 00:23:40.416 --> 00:23:44.300 And the third argument is the blend. 00:23:44.383 --> 00:23:49.783 So we have to give it an amount of each color or sort of 00:23:49.783 --> 00:23:53.800 where it should fall on the linear scale 00:23:53.800 --> 00:23:57.166 between this red and yellow. 00:23:57.250 --> 00:24:00.783 So this amount, if it's set to zero, 00:24:00.783 --> 00:24:03.933 we will just get the the from color. 00:24:03.933 --> 00:24:09.183 If it's set to one, then it will only get the to color. 00:24:09.433 --> 00:24:13.250 And if it's 0.5, then we'll get somewhere in the middle. 00:24:13.333 --> 00:24:14.833 So we are 00:24:14.833 --> 00:24:17.833 going to make this or we can make this 00:24:17.916 --> 00:24:22.066 just a random value somewhere in between the two. 00:24:22.150 --> 00:24:31.200 So somewhere between zero and one. 00:24:31.283 --> 00:24:33.550 Then we can 00:24:33.550 --> 00:24:35.533 call fill 00:24:35.533 --> 00:24:39.150 and the color that we want and fill is this lerp color. 00:24:39.166 --> 00:24:42.066 So see, 00:24:42.066 --> 00:24:43.033 and then I think 00:24:43.033 --> 00:24:47.150 we will keep the rest the same. 00:24:47.233 --> 00:24:52.783 Let's make the size of our circles slightly smaller. 00:24:52.866 --> 00:24:55.483 And otherwise, let's keep it the same. 00:24:55.483 --> 00:24:59.650 We also need to update that we are no longer drawing key points. 00:24:59.650 --> 00:25:04.266 We are drawing cold hands. 00:25:04.350 --> 00:25:05.116 And the last thing 00:25:05.116 --> 00:25:08.366 we should do before we run this 00:25:08.450 --> 00:25:14.516 is that we should remove our video image 00:25:14.600 --> 00:25:17.183 and we can replace 00:25:17.183 --> 00:25:22.683 this video image with a background 00:25:22.766 --> 00:25:23.900 instead. 00:25:23.900 --> 00:25:25.850 Let's just make this 00:25:25.850 --> 00:25:28.250 a black background for now. 00:25:28.250 --> 00:25:36.183 So let's run this and see what it looks like. 00:25:36.266 --> 00:25:38.483 So you can see now that the model is loaded. 00:25:38.483 --> 00:25:41.450 So if I put my hand in front, 00:25:41.450 --> 00:25:44.333 can see that we get 00:25:44.333 --> 00:25:47.333 the hand tracking. 00:25:47.550 --> 00:25:48.983 It's working 00:25:48.983 --> 00:25:52.750 fairly well cold colors. 00:25:52.816 --> 00:25:55.350 So let's move on to our other 00:25:55.350 --> 00:25:59.533 visualization of drawing hot hands. 00:25:59.616 --> 00:26:01.266 So in our hot 00:26:01.266 --> 00:26:04.866 visualization, we're going to do something quite similar. 00:26:04.866 --> 00:26:07.866 We're still doing this double for loop. 00:26:07.983 --> 00:26:11.050 So I'm going to still create 00:26:11.050 --> 00:26:14.633 a separate function. 00:26:14.700 --> 00:26:17.250 I'm going to rename this 00:26:17.250 --> 00:26:21.883 draw Hot 00:26:21.966 --> 00:26:24.583 Hands, because we're going to want to switch 00:26:24.583 --> 00:26:28.850 between these two at a later point. 00:26:28.933 --> 00:26:31.050 And here 00:26:31.050 --> 00:26:35.950 I obviously want to change my color, my lerp color C, 00:26:35.950 --> 00:26:40.066 to not go from cold to cold, but 00:26:40.150 --> 00:26:43.150 using my hot color. 00:26:43.183 --> 00:26:47.183 So hot color from. 00:26:47.250 --> 00:26:53.533 And hot color, too. 00:26:53.616 --> 00:26:55.650 And for 00:26:55.650 --> 00:27:00.500 this one, I think we can do something slightly different. 00:27:00.583 --> 00:27:04.150 Let's make the gradient depend on the landmark 00:27:04.150 --> 00:27:07.350 by setting our lerp amount to correspond to 00:27:07.600 --> 00:27:11.750 where in the list of landmark, the current key point is. 00:27:11.833 --> 00:27:15.216 And so we can do this by adding a local variable. 00:27:15.216 --> 00:27:20.800 Let's call it just A for amount. 00:27:20.883 --> 00:27:25.516 And we can set this to 00:27:25.600 --> 00:27:31.066 0.0 to start 00:27:31.150 --> 00:27:33.216 and remove our random. 00:27:33.216 --> 00:27:38.050 And instead we're going to use this amount A. 00:27:38.133 --> 00:27:40.000 And in order to make it depend 00:27:40.000 --> 00:27:43.500 on where in our landmarks 00:27:43.500 --> 00:27:47.633 list we are, we can turn our landmark indicators. 00:27:47.633 --> 00:27:50.283 So this j are in a for loop. 00:27:50.283 --> 00:27:53.283 We can turn that index value 00:27:53.283 --> 00:27:58.416 into a percentage so that it gives us 00:27:58.416 --> 00:28:02.500 sort of the percentage of how far we have gotten in the list. 00:28:02.583 --> 00:28:04.750 So we do that by 00:28:04.833 --> 00:28:08.750 adding j to our A value, 00:28:08.833 --> 00:28:12.516 but then dividing J by the list of this length. 00:28:12.516 --> 00:28:16.150 So we can just copy this 00:28:16.233 --> 00:28:21.200 value from our for loop. 00:28:21.283 --> 00:28:25.116 So I don't misspell anything there. 00:28:25.200 --> 00:28:29.016 By doing this, we get a sort of gradient 00:28:29.100 --> 00:28:34.266 going from the bottom of our hand and all the way up. 00:28:34.350 --> 00:28:39.816 We can also change from this ellipse 00:28:39.900 --> 00:28:43.616 and make them sort of more distinct from each other. 00:28:43.700 --> 00:28:48.233 For example, by instead of using this circle shape, 00:28:48.383 --> 00:28:54.000 we can draw a line so we have two lines can't really be filled. 00:28:54.083 --> 00:28:59.050 So we have to change our call to fill to equal to stroke. 00:28:59.116 --> 00:29:01.500 And we don't want no stroke 00:29:01.500 --> 00:29:05.683 because that's what we need when we're drawing a line. 00:29:05.766 --> 00:29:08.583 I'm going to use the line function. 00:29:08.666 --> 00:29:09.266 And as you 00:29:09.266 --> 00:29:12.266 might know, the line takes four arguments. 00:29:12.266 --> 00:29:15.600 So X and Y of the start point of the line and then X 00:29:15.600 --> 00:29:18.600 and y of the end point of the line. 00:29:18.683 --> 00:29:22.316 So our start point will be 00:29:22.400 --> 00:29:26.400 this location of our key 00:29:26.400 --> 00:29:30.566 points, our current key point. 00:29:30.650 --> 00:29:33.600 And oh, that went outside. 00:29:33.600 --> 00:29:34.833 Let's put that inside. 00:29:34.833 --> 00:29:37.850 Sometimes it's hard to really tell if you're inside or outside 00:29:37.850 --> 00:29:41.100 the parentheses. 00:29:41.183 --> 00:29:41.866 Yes. So our start point 00:29:41.866 --> 00:29:45.350 is the X and Y position of our key point. 00:29:45.350 --> 00:29:48.350 And then we can 00:29:48.416 --> 00:29:53.200 basically just if we want to draw a line straight upward, 00:29:53.266 --> 00:29:56.950 we can use the same 00:29:57.016 --> 00:30:00.950 x value for our second point 00:30:01.100 --> 00:30:04.100 the line should connect to and then we can 00:30:04.100 --> 00:30:07.800 for our Y, we can just add some pixels. 00:30:07.800 --> 00:30:12.300 So, 30 for example, you can make them longer or shorter 00:30:12.383 --> 00:30:14.483 depending on what your 00:30:14.483 --> 00:30:19.016 what your style is. 00:30:19.100 --> 00:30:23.200 So yeah, let's have a look at 00:30:23.266 --> 00:30:25.050 this hot hand's function. 00:30:25.050 --> 00:30:30.266 So let's just replace our call to cold hands to draw 00:30:30.350 --> 00:30:42.516 hot hands instead. 00:30:42.600 --> 00:30:44.700 So now the model is loaded. 00:30:44.700 --> 00:30:49.150 You can see how this gradient is different. 00:30:49.216 --> 00:30:54.116 This is also a way of sort of looking at 00:30:54.200 --> 00:30:57.450 the order of these landmarks. 00:30:57.450 --> 00:31:00.266 So since we know that the color from 00:31:00.266 --> 00:31:04.650 is the most red, we can see that it starts at the palm 00:31:04.733 --> 00:31:07.800 and that the pinky is the last 00:31:07.916 --> 00:31:13.366 one of the landmarks that are in our list. 00:31:13.450 --> 00:31:16.450 And another thing that I wanted to 00:31:16.600 --> 00:31:20.483 show you, just to make things look a little more interesting 00:31:20.550 --> 00:31:25.483 is that we can do a small hack with our call 00:31:25.483 --> 00:31:31.100 to background in our draw function, and we can 00:31:31.183 --> 00:31:31.650 give it 00:31:31.650 --> 00:31:35.533 an additional value, which is called an alpha value. 00:31:35.616 --> 00:31:40.733 And this basically makes the background slightly transparent out. 00:31:40.800 --> 00:31:47.016 So what happens then is we get this sort of trail effect 00:31:47.016 --> 00:31:50.650 because every time the draw function is called, which is 00:31:50.733 --> 00:31:55.216 I think 60 times per second, 00:31:55.300 --> 00:31:58.716 the background will draw over itself. 00:31:58.716 --> 00:32:02.383 And so we'll get a slight like trail behind it. 00:32:02.383 --> 00:32:08.333 We can have a look what it looks like. 00:32:08.400 --> 00:32:10.033 It's not my model is loaded. 00:32:10.033 --> 00:32:12.133 You can see my hand now. 00:32:12.133 --> 00:32:16.033 You can see that we get this slight cool looking delay. 00:32:16.200 --> 00:32:20.833 There are lots of very advanced ways of making a trail, but if you have something 00:32:21.066 --> 00:32:26.050 that is basically just drawing on a canvas, 00:32:26.133 --> 00:32:33.233 then this is a good hack for making making visual trails. 00:32:33.316 --> 00:32:35.866 So now let's take this a step further 00:32:35.866 --> 00:32:40.050 by including some image classification in the mix. 00:32:40.116 --> 00:32:43.783 So I'll be using the script from our third workshop, 00:32:43.783 --> 00:32:48.833 which was on image classification, which you can find on our websites 00:32:48.916 --> 00:32:50.600 to see the whole workshop. 00:32:50.600 --> 00:32:54.650 Or you can just use the code which is under the header of part 00:32:54.650 --> 00:32:57.133 four of this page. 00:32:57.133 --> 00:33:02.400 Now I won't spend too much time on this script going through it, 00:33:02.466 --> 00:33:05.866 but I will explain what we are doing and I'll also add 00:33:05.866 --> 00:33:10.616 some of this functionality to our hand script. 00:33:10.683 --> 00:33:16.716 So as you can see here, we have some buttons to do, some image 00:33:16.716 --> 00:33:22.200 classification training, a model, and we can also save and load this model. 00:33:22.216 --> 00:33:26.083 So let's try and build a classifier 00:33:26.083 --> 00:33:30.750 that can recognize some hand poses. 00:33:30.816 --> 00:33:31.300 So the 00:33:31.300 --> 00:33:35.033 sign language for cold is something like this. 00:33:35.050 --> 00:33:40.283 So with two hands now, we'll just be using one 00:33:40.366 --> 00:33:42.950 and for fire for our hot 00:33:42.950 --> 00:33:45.950 hands is something like this. 00:33:46.116 --> 00:33:49.450 And again, we'll be using just one hand, 00:33:49.450 --> 00:33:54.933 but we'll just have to approximate sorry to anyone who knows sign language. 00:33:55.016 --> 00:33:58.750 And in this script, 00:33:58.850 --> 00:34:02.366 we can train and save an image classifier. 00:34:02.450 --> 00:34:06.883 So we can train it to detect when we are showing a fist 00:34:06.883 --> 00:34:10.216 or when we are showing our palms. 00:34:10.300 --> 00:34:14.716 So this classifier has two classes, Class A and Class B, 00:34:14.800 --> 00:34:19.183 And so by clicking on these add image buttons, we are collecting 00:34:19.183 --> 00:34:24.633 images from our video feed to use in training the image classifier. 00:34:24.716 --> 00:34:30.316 We can collect around 30 images or so for Class A showing our close taste 00:34:30.316 --> 00:34:35.400 and we'll do the same for Class B, but holding our palms up. 00:34:35.466 --> 00:34:37.100 So let's start with our 00:34:37.100 --> 00:34:41.516 class A showing our closed fists. 00:34:41.600 --> 00:34:45.233 So it's useful to sort of move it around a little bit 00:34:45.316 --> 00:34:48.716 the same way we'll be doing 00:34:48.800 --> 00:34:51.583 afterwards just to give some variation 00:34:51.583 --> 00:34:55.100 and make the script or the model 00:34:55.183 --> 00:34:58.400 less sensitive to changes. 00:34:58.483 --> 00:35:03.066 So yeah, I clicked a lot of 70 images for our closed fist, 00:35:03.066 --> 00:35:09.833 and so I'll try and do around the same for our open PARM version. 00:35:09.916 --> 00:35:12.316 I want to do 00:35:12.316 --> 00:35:15.500 one to do a little bit of the wiggling and for some reason that maybe 00:35:15.500 --> 00:35:19.500 by clicking all weird 00:35:19.566 --> 00:35:23.000 clicks a little slower maybe to get this to work. 00:35:23.083 --> 00:35:27.050 I'm just trying to change around a little bit, 00:35:27.133 --> 00:35:29.833 maybe to this way. 00:35:29.833 --> 00:35:30.466 There we go. 00:35:30.466 --> 00:35:35.783 We have about the same number now so we can click train. 00:35:35.866 --> 00:35:38.866 The model should start training 00:35:39.050 --> 00:35:40.850 and now it says Done training. 00:35:40.850 --> 00:35:43.716 So let's before we save it, let's test that. 00:35:43.716 --> 00:35:47.000 It is actually doing somewhat what we want it to do. 00:35:47.116 --> 00:35:52.283 So when I have a fist. 00:35:52.366 --> 00:35:56.700 Oh, not so sure. 00:35:56.766 --> 00:35:59.266 Better when I hold it further down. 00:35:59.266 --> 00:36:01.266 It's good to know 00:36:01.266 --> 00:36:05.416 and it's very good at classifying my palm. 00:36:05.500 --> 00:36:09.083 So let's say that this is good enough for now at least. 00:36:09.166 --> 00:36:13.100 So then we can press save model. 00:36:13.183 --> 00:36:16.333 So save model will download two files 00:36:16.566 --> 00:36:21.716 and put them in our downloads folder 00:36:21.800 --> 00:36:24.583 and then we can 00:36:24.583 --> 00:36:28.916 shift back to our other script 00:36:29.000 --> 00:36:30.583 and let's implement 00:36:30.583 --> 00:36:35.916 some of this functionality for our hand pose. 00:36:35.983 --> 00:36:37.516 So now back in our handpose 00:36:37.516 --> 00:36:41.633 script we can add the load and the predict functionality 00:36:41.816 --> 00:36:45.966 from our image classifier and have it classify our hand poses. 00:36:46.033 --> 00:36:49.533 So to add the image classifier, we first need to declare 00:36:49.566 --> 00:36:54.833 the feature extractor that the model is built on. 00:36:54.916 --> 00:36:56.116 So let's do that. 00:36:56.116 --> 00:36:59.383 Underneath our hand pose, we'll declare 00:36:59.383 --> 00:37:02.383 a feature 00:37:02.416 --> 00:37:06.383 extractor 00:37:06.466 --> 00:37:08.083 and we'll get that feature 00:37:08.083 --> 00:37:11.083 extracted from ML5´s 00:37:11.116 --> 00:37:16.383 feature extract. 00:37:16.466 --> 00:37:19.650 And they have a couple of different ones 00:37:19.733 --> 00:37:23.116 that we can use, but we should keep these definitions 00:37:23.116 --> 00:37:28.550 the same as for the classifier that we just trained. 00:37:28.583 --> 00:37:31.916 So that one was used 00:37:32.000 --> 00:37:35.850 a model called mobilenet. 00:37:35.933 --> 00:37:38.116 So let's do the same. 00:37:38.116 --> 00:37:41.516 And again, we should give it 00:37:41.516 --> 00:37:45.300 a callback function to let us know when the model is ready. 00:37:45.383 --> 00:37:49.533 So now we already have a model ready function. 00:37:49.616 --> 00:37:54.750 So let's specify that this one is for mobilenet. 00:37:54.833 --> 00:37:57.500 So we'll see that mobilenet is loaded. 00:37:57.500 --> 00:38:00.783 And then let's change the hand pose one. 00:38:00.866 --> 00:38:03.283 Just give it a HP in front. 00:38:03.283 --> 00:38:07.416 So it's now HP hand pose model ready 00:38:07.483 --> 00:38:13.016 and we'll make a similar function for hand pose. 00:38:13.100 --> 00:38:15.566 So that's HP model ready. 00:38:15.566 --> 00:38:20.266 And this one should then say hand pose is loaded. 00:38:20.333 --> 00:38:25.566 We now can go back to our setup function under our feature extractor. 00:38:25.650 --> 00:38:30.816 We're going to define some options for our actual image classifier, 00:38:30.816 --> 00:38:34.716 which is built from the features of mobile net. 00:38:34.800 --> 00:38:41.200 So what we're going to do then is we're going to define a constant called options, 00:38:41.283 --> 00:38:44.283 which should be an object 00:38:44.466 --> 00:38:50.950 that uses the the tags that are in 00:38:51.033 --> 00:38:52.983 the ML5 image classifier. 00:38:52.983 --> 00:38:56.733 So one is called num labels. 00:38:56.816 --> 00:38:59.800 We want to set that 00:38:59.800 --> 00:39:03.916 to colon and set that to two 00:39:04.016 --> 00:39:11.116 because we trained a model with two different classes, two labels. 00:39:11.183 --> 00:39:13.350 So that's our options. 00:39:13.350 --> 00:39:16.900 And now we can define our image classifier model. 00:39:16.983 --> 00:39:19.500 So we can write that model equals. 00:39:19.500 --> 00:39:26.950 We'll use this feature extractor that we defined. 00:39:27.033 --> 00:39:27.450 I'm going to 00:39:27.450 --> 00:39:31.333 use its classification, but there should be a dot. 00:39:31.350 --> 00:39:35.316 Sorry, 00:39:35.400 --> 00:39:39.783 it's classification method 00:39:39.866 --> 00:39:45.883 and we need to give it access to our video 00:39:45.966 --> 00:39:50.433 and our options 00:39:50.516 --> 00:39:53.400 like so. 00:39:53.400 --> 00:39:54.050 So we'll also 00:39:54.050 --> 00:39:57.850 need to make some changes to our index HTML file 00:39:57.933 --> 00:40:02.550 to add some buttons and some information about loading 00:40:02.633 --> 00:40:06.750 for actually getting access to this Pre-trained model. 00:40:06.750 --> 00:40:13.300 So let's go into our sketch files and move to index dot each HTML. 00:40:13.383 --> 00:40:22.000 We're going to add these buttons inside our body here 00:40:22.083 --> 00:40:26.550 and then we will do something like this 00:40:26.633 --> 00:40:30.216 button with the ID, 00:40:30.300 --> 00:40:37.683 let's call it button predicts. 00:40:37.766 --> 00:40:40.766 And we want that button to say, 00:40:40.916 --> 00:40:43.350 let's start 00:40:43.350 --> 00:40:44.150 guessing. 00:40:44.150 --> 00:40:49.783 So now just copying this directly from our previous project, 00:40:49.866 --> 00:40:55.183 then we and our button definition 00:40:55.250 --> 00:41:05.016 and let's add a line break. 00:41:05.100 --> 00:41:10.983 We also want a separate paragraph, 00:41:11.066 --> 00:41:16.983 so we use P for that to set up our load load function. 00:41:17.066 --> 00:41:23.883 So we'll then use the HTML tag label 00:41:23.966 --> 00:41:32.083 for equals 00:41:32.150 --> 00:41:35.783 Avatar. 00:41:35.850 --> 00:41:38.600 We want 00:41:38.600 --> 00:41:43.533 this button to say or this 00:41:43.616 --> 00:41:48.400 module to say load model, 00:41:48.483 --> 00:41:57.300 and that's the end of our label. 00:41:57.383 --> 00:42:01.750 It's the way around sorry. 00:42:01.833 --> 00:42:03.833 And then we should define 00:42:03.833 --> 00:42:08.633 what kind of files that we can receive in our load function. 00:42:08.633 --> 00:42:16.483 So we have to give it an input 00:42:16.550 --> 00:42:23.766 and the input will be of type A file 00:42:23.850 --> 00:42:25.250 and we'll give it an ID 00:42:25.250 --> 00:42:29.700 so that we can access it from our 00:42:29.783 --> 00:42:31.466 P5 script. 00:42:31.466 --> 00:42:34.800 So let's call this load 00:42:34.883 --> 00:42:36.750 and we'll specify that 00:42:36.750 --> 00:42:41.383 we want to be able to load multiple files. 00:42:41.450 --> 00:42:45.583 Then we can end our input statement 00:42:45.650 --> 00:42:53.183 and remember to end our paragraph. 00:42:53.266 --> 00:42:55.333 So back in our script, 00:42:55.333 --> 00:42:58.333 our sketch dot JS file, 00:42:58.400 --> 00:43:03.683 we can now implement these buttons in our setup function. 00:43:03.766 --> 00:43:06.383 So underneath our color 00:43:06.383 --> 00:43:12.083 definitions, the end of our setup function, we can define 00:43:12.166 --> 00:43:15.900 our load button as just a 00:43:15.983 --> 00:43:19.350 load button. 00:43:19.433 --> 00:43:20.783 And to access 00:43:20.783 --> 00:43:25.283 these HTML elements, we'll use select 00:43:25.366 --> 00:43:28.983 and then refer to this tag that we gave it. 00:43:29.066 --> 00:43:33.066 So for load but not tag was or our ID was load. 00:43:33.133 --> 00:43:38.533 So select load 00:43:38.616 --> 00:43:40.616 And for the load button we're going to use the P5 00:43:40.616 --> 00:43:45.216 function changed, which listens to changes in the buttons. 00:43:45.216 --> 00:43:50.683 So the way we do that is we say load 00:43:50.766 --> 00:43:55.600 button dot changed 00:43:55.683 --> 00:43:59.633 and in here we're going to do sort of a compact 00:43:59.633 --> 00:44:04.300 function to deal with how this load button is supposed to work. 00:44:04.383 --> 00:44:09.816 So we're going to define our function directly in here, 00:44:09.900 --> 00:44:15.700 taking no arguments and 00:44:15.783 --> 00:44:19.116 we want the load button 00:44:19.200 --> 00:44:24.816 to call our model dot load. 00:44:24.900 --> 00:44:27.050 And luckily ML5, 00:44:27.050 --> 00:44:32.200 the image classifier has already a lot of this functionality 00:44:32.283 --> 00:44:33.666 already done for us. 00:44:33.666 --> 00:44:38.833 But what we need to do is we need to get the files 00:44:38.916 --> 00:44:42.316 that we will be supplying through a load button. 00:44:42.383 --> 00:44:47.683 So we will do load button 00:44:47.766 --> 00:44:52.716 elt dot files 00:44:52.800 --> 00:44:57.233 and we are going to do another nested 00:44:57.416 --> 00:45:01.150 function call here 00:45:01.233 --> 00:45:05.716 which will just log to our console 00:45:05.783 --> 00:45:13.900 that our custom model has been loaded. 00:45:13.983 --> 00:45:21.416 This now 00:45:21.416 --> 00:45:24.950 predicts button is a bit simpler to define. 00:45:24.950 --> 00:45:27.066 So let's just hit a button. 00:45:27.066 --> 00:45:30.616 Predict 00:45:30.683 --> 00:45:33.233 equals. 00:45:33.233 --> 00:45:38.433 We need to select again our predict button, 00:45:38.516 --> 00:45:48.333 which we gave the tag predict. 00:45:48.416 --> 00:45:51.416 Did we not, let me check. 00:45:51.483 --> 00:45:56.316 Button predict was the tag that we gave it. 00:45:56.400 --> 00:45:59.883 So hash tag, button predict. 00:45:59.966 --> 00:46:01.733 And then 00:46:01.733 --> 00:46:04.350 we basically just want it 00:46:04.350 --> 00:46:08.866 to react to being pressed 00:46:08.933 --> 00:46:09.866 button predict 00:46:09.866 --> 00:46:15.616 Button predict 00:46:15.683 --> 00:46:22.033 Mouse pressed 00:46:22.116 --> 00:46:24.233 and if it detects 00:46:24.233 --> 00:46:28.416 that the mouse has been pressed on this button, 00:46:28.500 --> 00:46:30.833 we want it to call a classify function. 00:46:30.833 --> 00:46:35.033 So we're going to define what this function classify does. 00:46:35.033 --> 00:46:38.650 So outside of our setup function, 00:46:38.733 --> 00:46:40.866 we're going to create our own function 00:46:40.866 --> 00:46:44.616 called classify. 00:46:44.700 --> 00:46:48.450 And this in turn will be in charge 00:46:48.450 --> 00:46:54.400 of calling our image classifiers or classify function. 00:46:54.483 --> 00:47:00.250 And to do that, we should also 00:47:00.333 --> 00:47:04.383 give it as an argument, a callback function, 00:47:04.383 --> 00:47:12.883 which we also want to define, which will call got results. 00:47:12.966 --> 00:47:14.633 And so this got results 00:47:14.633 --> 00:47:19.233 function will implement a logic similar 00:47:19.233 --> 00:47:23.883 to what we used in our Image Classification workshop. 00:47:23.966 --> 00:47:27.483 So we'll be defining two boolean values 00:47:27.483 --> 00:47:31.416 hot and cold and setting them both to false at the start. 00:47:31.500 --> 00:47:34.866 And then we want to deal with what 00:47:34.950 --> 00:47:38.100 we set these Boolean values to based on 00:47:38.100 --> 00:47:42.366 what kind of results we get from our image classifier. 00:47:42.450 --> 00:47:44.683 So let's define function. 00:47:44.683 --> 00:47:51.133 Got results. 00:47:51.216 --> 00:47:55.383 And as you may remember, we can get sort of two different kinds of results. 00:47:55.383 --> 00:48:00.066 We can get an error or we can actually get results. 00:48:00.150 --> 00:48:01.733 So let's handle both of them. 00:48:01.733 --> 00:48:05.350 So if it's error, 00:48:05.433 --> 00:48:12.216 we want to just get our console 00:48:12.300 --> 00:48:15.116 to log that error for us 00:48:15.116 --> 00:48:18.433 so we can see what it is. 00:48:18.516 --> 00:48:20.166 And otherwise 00:48:20.166 --> 00:48:24.033 we want to have a look at these results. 00:48:24.116 --> 00:48:27.000 So I'm not going to go into depth on what these results 00:48:27.000 --> 00:48:30.683 look like to know sort of more about this image classification model. 00:48:30.683 --> 00:48:34.433 You can always check out the other scripts to see it 00:48:34.433 --> 00:48:38.133 more in detail or the other video from our previous workshop. 00:48:38.216 --> 00:48:41.383 So for now we're just going to do 00:48:41.466 --> 00:48:42.383 let's see, we're 00:48:42.383 --> 00:48:45.383 going to do below this, going to say if 00:48:45.450 --> 00:48:49.933 we have results 00:48:50.016 --> 00:48:52.716 and we want to access 00:48:52.716 --> 00:48:55.083 results, 00:48:55.083 --> 00:48:57.450 the first value in our results, 00:48:57.450 --> 00:49:02.066 which is going to be the value with the highest confidence. 00:49:02.066 --> 00:49:05.850 So basically what the model is predicting is the correct class 00:49:05.850 --> 00:49:10.116 for a given frame. 00:49:10.200 --> 00:49:12.783 Then we are going to 00:49:12.783 --> 00:49:15.750 look specifically at what that value is. 00:49:15.750 --> 00:49:17.366 What is that result zero. 00:49:17.366 --> 00:49:22.366 What is the most likely predicted class? 00:49:22.433 --> 00:49:25.133 And so if 00:49:25.133 --> 00:49:27.750 results 00:49:27.750 --> 00:49:32.916 zero is equal to our first class, 00:49:33.000 --> 00:49:35.733 Class A, 00:49:35.733 --> 00:49:38.733 then we want 00:49:38.733 --> 00:49:40.883 our boolean value hot, 00:49:40.883 --> 00:49:44.883 which will define to become true 00:49:44.966 --> 00:49:47.966 and our 00:49:48.216 --> 00:49:55.116 boolean value called to become false. 00:49:55.200 --> 00:49:58.733 And likewise, we want to check for the other way around. 00:49:58.733 --> 00:50:01.416 So if the result in place zero. 00:50:01.416 --> 00:50:04.683 So the most likely result is B, 00:50:04.766 --> 00:50:08.400 then we want hot to turn 00:50:08.400 --> 00:50:13.416 false and cold to turn true. 00:50:13.500 --> 00:50:16.750 Now we can define 00:50:16.833 --> 00:50:19.616 these variables hot 00:50:19.616 --> 00:50:23.916 and cold at the outset of our program 00:50:24.000 --> 00:50:34.716 and set them both to false from the get go. 00:50:34.800 --> 00:50:38.833 So the last thing that we want to do is 00:50:38.900 --> 00:50:40.950 call classify again. 00:50:40.950 --> 00:50:45.816 So if we have some results 00:50:45.900 --> 00:50:50.316 yet, so we need to actually move this 00:50:50.400 --> 00:50:53.450 into our second check here. 00:50:53.450 --> 00:50:55.800 If we have results. 00:50:55.800 --> 00:50:56.966 There has been no error. 00:50:56.966 --> 00:51:00.333 We want to keep classifying. 00:51:00.416 --> 00:51:02.933 So before we update our draw function, 00:51:02.933 --> 00:51:08.533 we can also put this new model variable, make that global 00:51:08.533 --> 00:51:12.483 so that P5 doesn't confuse it with any other 00:51:12.566 --> 00:51:14.866 definitions of model. 00:51:14.866 --> 00:51:17.083 And then finally, let's change 00:51:17.083 --> 00:51:21.233 our draw function to check 00:51:21.316 --> 00:51:27.833 if our cold value is true. 00:51:27.916 --> 00:51:29.933 If it is, 00:51:29.933 --> 00:51:33.866 we want to call the draw 00:51:33.950 --> 00:51:36.533 cold hands 00:51:36.533 --> 00:51:40.416 function that we defined earlier, 00:51:40.483 --> 00:51:47.283 and if hot is true, 00:51:47.366 --> 00:51:51.866 we want to draw our hot hands 00:51:51.950 --> 00:51:57.483 should already have here. 00:51:57.566 --> 00:51:59.000 There we go. 00:51:59.000 --> 00:52:03.933 Let's make this a little more readable 00:52:04.016 --> 00:52:07.433 so now we can load our model files 00:52:07.433 --> 00:52:10.933 and test this out. 00:52:11.016 --> 00:52:15.316 So let's press play and see what happens. 00:52:15.400 --> 00:52:17.916 So we get our black background. 00:52:17.916 --> 00:52:23.033 Mobile net is loaded and hand pose is loaded, so we should be ready to go. 00:52:23.116 --> 00:52:29.216 So let's load our mobile files. 00:52:29.283 --> 00:52:32.116 So now we see there are two files selected. 00:52:32.116 --> 00:52:33.483 We can press start guessing 00:52:33.483 --> 00:52:35.800 00:52:35.800 --> 00:52:40.433 and we should see a hand, but we don't. 00:52:40.516 --> 00:52:42.666 So let's see if we can figure out 00:52:42.666 --> 00:52:45.666 why that might be. 00:52:45.916 --> 00:52:47.950 Because we know that this model 00:52:47.950 --> 00:52:53.533 was able to recognize something. 00:52:53.616 --> 00:52:54.366 Yeah. 00:52:54.366 --> 00:52:56.200 So for 00:52:56.200 --> 00:53:00.166 those of you who have finished our image classification 00:53:00.166 --> 00:53:04.766 workshop might already have caught what this error was. 00:53:04.833 --> 00:53:08.116 Our results from our image classifier 00:53:08.200 --> 00:53:11.233 actually gives us both a label, 00:53:11.233 --> 00:53:16.483 which is the value that it's predicting and also a confidence value. 00:53:16.566 --> 00:53:20.016 So within this results zero, we actually have two attributes 00:53:20.016 --> 00:53:25.033 both confidence and label, and it's label that we are interested in. 00:53:25.116 --> 00:53:29.600 So we have to do result zero dot label check if that's A 00:53:29.683 --> 00:53:32.683 or B, 00:53:32.683 --> 00:53:36.683 And so let's try this again. 00:53:36.766 --> 00:53:38.683 So we press play. 00:53:38.683 --> 00:53:42.116 We wait for our models to be loaded. 00:53:42.183 --> 00:53:46.483 Now we can select our model files 00:53:46.566 --> 00:53:47.900 and start guessing. 00:53:47.900 --> 00:53:49.683 So now we should at least see a hand. 00:53:49.683 --> 00:53:51.633 And we do. 00:53:51.633 --> 00:53:53.866 Okay, good. 00:53:53.866 --> 00:53:56.316 But it seems 00:53:56.316 --> 00:53:59.666 to be the wrong way around. 00:53:59.733 --> 00:54:02.383 So this is our cold hand. 00:54:02.383 --> 00:54:04.116 And this is our fist. 00:54:04.116 --> 00:54:05.616 Is our hot hand. 00:54:05.616 --> 00:54:09.583 But that's luckily an easy fix. 00:54:09.666 --> 00:54:13.666 So we want to change this to true 00:54:13.750 --> 00:54:16.933 this, to false, 00:54:17.016 --> 00:54:19.383 this to false. 00:54:19.383 --> 00:54:20.050 And this 00:54:20.050 --> 00:54:23.966 to true, So basically just switching them around. 00:54:24.033 --> 00:54:31.816 Let's try one last time 00:54:31.900 --> 00:54:34.266 and start guessing. 00:54:34.266 --> 00:54:36.183 There we go. 00:54:36.183 --> 00:54:38.500 There's our little fire symbol 00:54:38.500 --> 00:54:41.933 and our coldest. 00:54:42.016 --> 00:54:43.983 And that's it for now. 00:54:43.983 --> 00:54:48.200 And so you can obviously take this further in many ways. 00:54:48.283 --> 00:54:51.616 Some examples are on our website 00:54:51.616 --> 00:54:54.616 under the header going further. 00:54:54.766 --> 00:54:56.783 So thank you very much for joining.