WEBVTT 00:00:08.500 --> 00:00:10.600 Hello and welcome back to the 00:00:10.600 --> 00:00:14.666 third C2HO workshop focusing on an introduction 00:00:14.666 --> 00:00:19.000 to machine learning for creative Computing with ML5. 00:00:19.083 --> 00:00:21.783 The ML5 library allows us to utilize machine learning 00:00:21.783 --> 00:00:25.933 directly in our web browser using P5.js. 00:00:26.016 --> 00:00:29.100 and the ML5 Library also gives us access to a bunch 00:00:29.100 --> 00:00:32.500 of Pre-trained models, and today we will be playing around 00:00:32.500 --> 00:00:37.516 with image classification using the mobilenet model. 00:00:37.600 --> 00:00:40.300 Mobilenet is a class of efficient models 00:00:40.300 --> 00:00:43.666 for mobile and embedded vision applications. 00:00:43.750 --> 00:00:48.816 It's designed with mobile devices in mind to support classification, detection 00:00:48.816 --> 00:00:53.933 and more and mobilenet is based on a streamlined architecture 00:00:54.000 --> 00:00:59.900 that uses lightweight, deep neural networks and trades off accuracy 00:00:59.983 --> 00:01:03.950 for low latency and a small memory footprint. 00:01:04.033 --> 00:01:08.383 And today we'll be providing training data through our webcams. 00:01:08.383 --> 00:01:13.033 So remember to allow your browser to access your webcam 00:01:13.033 --> 00:01:14.450 if you want to follow along 00:01:14.450 --> 00:01:19.400 with this presentation or run the code on your own machine. 00:01:19.483 --> 00:01:20.633 So as usual, I'll 00:01:20.633 --> 00:01:24.566 start by setting up my canvas. 00:01:24.650 --> 00:01:27.950 And since the width and height 00:01:27.950 --> 00:01:31.833 of our canvases will sort of 00:01:31.900 --> 00:01:33.850 change based on what kind of or 00:01:33.850 --> 00:01:36.850 what size of screen you are working on, 00:01:37.066 --> 00:01:41.000 I like to make these variables 00:01:41.083 --> 00:01:44.366 globally declared so that you can change them 00:01:44.450 --> 00:01:47.050 to fit your needs. 00:01:47.050 --> 00:01:51.783 So let's set our canvas height 00:01:51.850 --> 00:01:53.933 and canvas width. 00:01:53.933 --> 00:01:58.383 We can do that on the same line. 00:01:58.466 --> 00:02:03.200 So width and height. 00:02:03.283 --> 00:02:05.500 And we are going to use them 00:02:05.500 --> 00:02:08.933 instead of these 00:02:09.016 --> 00:02:13.850 values here. 00:02:13.933 --> 00:02:18.283 And we can just give them a value directly in our setup function, 00:02:18.283 --> 00:02:23.283 which will run before anything else in our script today. 00:02:23.350 --> 00:02:27.933 So let's set canvas 00:02:28.000 --> 00:02:31.316 width. 00:02:31.400 --> 00:02:40.800 Let's set it to 400 to start. 00:02:40.883 --> 00:02:42.400 And again, just change these 00:02:42.400 --> 00:02:47.566 if this doesn't really suit your needs, you can press play now and see 00:02:47.566 --> 00:02:51.333 what kind of a size that is what it looks like on your end. 00:02:51.400 --> 00:02:53.783 So we'll begin by setting up a video 00:02:53.783 --> 00:02:58.283 element using the P5 function create capture. 00:02:58.366 --> 00:03:02.250 So we will do that in our setup function. 00:03:02.333 --> 00:03:05.683 We will write video equals 00:03:05.833 --> 00:03:12.183 create capture, 00:03:12.250 --> 00:03:14.350 and we'll use 00:03:14.350 --> 00:03:18.233 the default video settings. 00:03:18.316 --> 00:03:22.466 So we just use the variable video 00:03:22.550 --> 00:03:26.283 and we'll give it a size, 00:03:26.350 --> 00:03:32.633 which is equal to or canvas. 00:03:32.716 --> 00:03:46.283 Let's see. 00:03:46.366 --> 00:03:50.216 And canvas height. 00:03:50.300 --> 00:03:51.200 Now, since 00:03:51.200 --> 00:03:54.833 we will be working mostly with video today, 00:03:54.833 --> 00:03:59.333 we don't really need to display this gray canvas. 00:03:59.333 --> 00:04:03.416 Instead, we will be displaying just our video. 00:04:03.500 --> 00:04:10.766 So we can turn off the canvas just by saying no canvas. 00:04:10.850 --> 00:04:13.116 Like so. 00:04:13.116 --> 00:04:14.850 So when we now press play, 00:04:14.850 --> 00:04:19.666 we should see our video popping up. 00:04:19.733 --> 00:04:21.700 And there it is. 00:04:21.700 --> 00:04:25.350 Now let's define our model and make sure that it's loading. 00:04:25.433 --> 00:04:29.633 So what we will be doing to build our image classifier 00:04:29.716 --> 00:04:35.183 is actually extracting the features from the Pre-trained mobilenet model 00:04:35.266 --> 00:04:40.433 and then simply retraining the last layers of this model to classify what we want. 00:04:40.516 --> 00:04:44.000 This is known as transfer learning because we are transferring 00:04:44.183 --> 00:04:47.800 most of what mobilenet has already learned 00:04:47.950 --> 00:04:51.483 into a new bespoke classification of problem. 00:04:51.550 --> 00:04:54.916 And this is good because usually deep neural networks like those used in image 00:04:54.916 --> 00:04:58.616 classification are very large and take a long time to train. 00:04:58.700 --> 00:05:02.983 But by extracting the features, we can benefit from this intensive 00:05:02.983 --> 00:05:06.016 training while still tailoring the model to 00:05:06.016 --> 00:05:09.866 whatever image classification task we see fit. 00:05:09.950 --> 00:05:15.683 So let's start by initializing our feature extractor 00:05:15.766 --> 00:05:20.466 so we can do feature 00:05:20.533 --> 00:05:23.783 extractor. 00:05:23.866 --> 00:05:25.616 00:05:25.616 --> 00:05:28.616 equals ML5 dot 00:05:28.816 --> 00:05:35.850 feature extractor 00:05:35.933 --> 00:05:39.233 and ML5 five has a couple of different models. 00:05:39.233 --> 00:05:42.983 Pre-trained models that we can extract features from. 00:05:43.066 --> 00:05:46.066 But today we will be using mobilenet. 00:05:46.166 --> 00:05:47.750 So we type that in. 00:05:47.750 --> 00:05:54.466 And we also want to be noticed of when this model has successfully loaded. 00:05:54.466 --> 00:05:57.466 So we're going to use a callback function, 00:05:57.466 --> 00:06:00.050 which is called Model 00:06:00.050 --> 00:06:02.900 Ready, which we will define. 00:06:02.900 --> 00:06:05.033 So we can do that 00:06:05.033 --> 00:06:06.083 just right away. 00:06:06.083 --> 00:06:08.416 So function 00:06:08.416 --> 00:06:12.716 model, ready. 00:06:12.800 --> 00:06:16.800 And we just want it to print 00:06:16.883 --> 00:06:18.550 the status of the model 00:06:18.550 --> 00:06:23.100 to our console window, which is down here to console.log. 00:06:23.183 --> 00:06:34.983 And we just want it to say that the model is ready. 00:06:35.066 --> 00:06:39.566 So we can now press play and see what that looks like. 00:06:39.650 --> 00:06:40.316 We get our 00:06:40.316 --> 00:06:45.733 video and we get this small notification that the model is ready. 00:06:45.816 --> 00:06:49.866 So let's stop this and 00:06:49.950 --> 00:06:53.983 we can now go back into our setup function 00:06:54.066 --> 00:06:58.066 and we can create a new classifier based on these features. 00:06:58.133 --> 00:07:01.266 So to do this, we need to give it our video as input 00:07:01.266 --> 00:07:05.250 and we need to define some settings for the model, mainly how many classes 00:07:05.250 --> 00:07:09.016 we want it to be able to differentiate between. 00:07:09.083 --> 00:07:13.933 So we will first define some options as constant 00:07:14.016 --> 00:07:17.833 options, 00:07:17.916 --> 00:07:21.966 and we'll use the keyword for mobilenet 00:07:22.050 --> 00:07:26.266 for defining how many classes, which is num 00:07:26.333 --> 00:07:29.333 labels, so number of labels. 00:07:29.433 --> 00:07:32.183 We set that to two. 00:07:32.183 --> 00:07:36.433 That's in these curly brackets. 00:07:36.516 --> 00:07:39.383 And then we can define our model 00:07:39.383 --> 00:07:57.100 as using this feature extractor that we defined. 00:07:57.183 --> 00:07:59.633 And we want to access 00:07:59.633 --> 00:08:02.633 the classification version 00:08:02.700 --> 00:08:05.983 of these extracted features. 00:08:06.066 --> 00:08:11.083 We'll send it our video element, see video 00:08:11.150 --> 00:08:15.616 and these options that we defined above. 00:08:15.683 --> 00:08:19.683 Now we want this model to be available throughout our program. 00:08:19.683 --> 00:08:23.766 So we will define it globally 00:08:23.850 --> 00:08:27.883 at the very top. 00:08:27.966 --> 00:08:31.966 So what we are going to be doing today is we're going to be 00:08:32.050 --> 00:08:37.066 training our image classifier live in the browser. 00:08:37.150 --> 00:08:41.533 So in our previous ML5 workshops, 00:08:41.533 --> 00:08:45.916 we've used our canvas to display and input data. 00:08:46.000 --> 00:08:51.550 But here we'll be adding buttons directly to our index.html file instead. 00:08:51.633 --> 00:08:55.133 And this will just make our scripts a little bit more readable 00:08:55.200 --> 00:08:59.233 because we're going to have a fair amount of buttons and labels. 00:08:59.316 --> 00:09:03.666 So to access and update the variables in our index HTML file, 00:09:03.883 --> 00:09:07.633 we'll use this P5 function select inside our script. 00:09:07.716 --> 00:09:12.233 So let's begin by adding some buttons to our index.html file. 00:09:12.300 --> 00:09:15.233 00:09:15.300 --> 00:09:19.483 HTML is notoriously not a pretty language, 00:09:19.683 --> 00:09:22.733 but hopefully you can follow along if you want to skip this part. 00:09:22.750 --> 00:09:26.416 You can also use the 00:09:26.500 --> 00:09:29.500 the P5 editor code that's available on our web pages. 00:09:29.533 --> 00:09:34.333 So you don't need to do all this scripting. 00:09:34.416 --> 00:09:36.816 But I will have to do it. 00:09:36.816 --> 00:09:38.100 So let's see. 00:09:38.100 --> 00:09:42.383 Inside our body, 00:09:42.450 --> 00:09:45.733 we can define some buttons. 00:09:45.816 --> 00:09:49.500 So let's create first off, 00:09:49.500 --> 00:09:53.600 two buttons for gathering some data 00:09:53.683 --> 00:09:55.800 since we have two classes is what we defined. 00:09:55.800 --> 00:10:00.066 We want to have a button for Class A and a button for Class B 00:10:00.066 --> 00:10:04.183 so let's start by setting those up. 00:10:04.266 --> 00:10:07.266 So we will define a button 00:10:07.333 --> 00:10:11.150 and it wouldn't have a ID 00:10:11.233 --> 00:10:17.200 and let's see, the ID can just be button A, 00:10:17.283 --> 00:10:22.533 This is what we'll use inside our script to sort of access these variables. 00:10:22.600 --> 00:10:23.916 We close 00:10:23.916 --> 00:10:27.533 Our brackets and we can add some text. 00:10:27.600 --> 00:10:32.100 So add image for class A, 00:10:32.183 --> 00:10:36.950 and then we need to close this 00:10:37.033 --> 00:10:41.183 this button description 00:10:41.250 --> 00:10:43.716 like so 00:10:43.716 --> 00:10:47.383 then we can add an identical button 00:10:47.466 --> 00:10:53.450 but change the label to be so ID is button B 00:10:53.533 --> 00:10:55.833 and our description should reflect that. 00:10:55.833 --> 00:10:59.566 So add image for class B, 00:10:59.650 --> 00:11:03.966 Now for these buttons, we also want to have a variable 00:11:03.966 --> 00:11:07.366 that will be counting 00:11:07.450 --> 00:11:10.350 how many examples we have collected. 00:11:10.350 --> 00:11:15.533 Basically how many times we've pressed these button A or B, So 00:11:15.600 --> 00:11:22.216 in order to access that, we use a tag which is called Span 00:11:22.300 --> 00:11:25.750 again, will give it an I.D. 00:11:25.833 --> 00:11:31.750 and let's call it a mount 00:11:31.833 --> 00:11:39.016 of class, A, 00:11:39.100 --> 00:11:42.850 and we need to give it an initial value, which should be zero. 00:11:42.850 --> 00:11:52.933 And then we can close this span 00:11:53.016 --> 00:11:55.150 and we can give it 00:11:55.150 --> 00:12:00.683 some text class a images. 00:12:00.750 --> 00:12:04.516 So in the beginning it'll just say zero class A images. 00:12:04.600 --> 00:12:08.416 And then as we keep adding images, 00:12:08.500 --> 00:12:12.466 we can increment this value. 00:12:12.550 --> 00:12:16.400 Then we will do a break, 00:12:16.466 --> 00:12:22.333 which just gives us a line break 00:12:22.416 --> 00:12:24.300 like so. 00:12:24.300 --> 00:12:27.083 And again, I'm just going to copy this and place 00:12:27.083 --> 00:12:31.650 it underneath Button B 00:12:31.733 --> 00:12:34.733 because I want the exact same functionality, 00:12:34.766 --> 00:12:39.666 but I'm going to change it from class A to B, 00:12:39.750 --> 00:12:41.666 But we also want this one to start from zero 00:12:41.666 --> 00:12:45.566 and so on, so we can just keep it like this. 00:12:45.633 --> 00:12:47.466 And the last thing I want to add 00:12:47.466 --> 00:12:52.816 is some information about the predictions 00:12:52.816 --> 00:12:55.816 that our image classifier is going to be making. 00:12:55.816 --> 00:12:59.600 So I want to be able to display that. 00:12:59.683 --> 00:13:06.500 So let's add a 00:13:06.583 --> 00:13:07.416 field for 00:13:07.416 --> 00:13:11.200 the labels and we want to be able to access this. 00:13:11.200 --> 00:13:19.633 So again, I'm going to use Span to set up a variable 00:13:19.716 --> 00:13:25.383 and it will just be a result. 00:13:25.450 --> 00:13:26.800 for now 00:13:26.800 --> 00:13:30.350 the default value can just be dot, dot, dot 00:13:30.433 --> 00:13:36.766 and then we can close this span 00:13:36.850 --> 00:13:39.183 and we can add a line break again 00:13:39.183 --> 00:13:44.150 for readability 00:13:44.233 --> 00:13:48.083 and we want to see the confidence. 00:13:48.166 --> 00:13:51.550 So basically, how confident 00:13:51.550 --> 00:13:56.266 the model is that its current prediction is correct? 00:13:56.350 --> 00:14:00.100 So we have our label, like what label it has predicted 00:14:00.100 --> 00:14:03.550 are Class A or copy and the confidence in that prediction. 00:14:03.550 --> 00:14:10.100 So again, I'm going to use span 00:14:10.183 --> 00:14:16.483 and our ID will be confidence 00:14:16.566 --> 00:14:18.916 and close that one. 00:14:18.916 --> 00:14:21.950 The default value can just be dot, dot, dot. 00:14:22.033 --> 00:14:27.950 And finally we close our span 00:14:28.033 --> 00:14:31.300 and close the element. 00:14:31.383 --> 00:14:38.700 So now let's go back to our sketch file 00:14:38.783 --> 00:14:43.433 and here we can add a new function button set up which will be in charge 00:14:43.433 --> 00:14:46.883 of changing the displayed values and handling 00:14:46.883 --> 00:14:50.366 what should happen when our buttons are pressed. 00:14:50.450 --> 00:14:53.666 So let's define that just under set up 00:14:53.750 --> 00:14:56.750 function, 00:14:56.900 --> 00:15:04.416 let's call it button set up. 00:15:04.483 --> 00:15:07.333 Now, as I mentioned, we want to be able to count 00:15:07.333 --> 00:15:10.466 how many examples of each class we have added. 00:15:10.466 --> 00:15:14.850 So let's declare to counter variables at the top of our script. 00:15:14.933 --> 00:15:18.666 So these will just count for us. So 00:15:18.733 --> 00:15:21.966 let's say Class A 00:15:22.033 --> 00:15:26.433 and we want them to initially be set to zero 00:15:26.516 --> 00:15:29.266 and Class B 00:15:29.266 --> 00:15:32.266 also set to zero. 00:15:32.366 --> 00:15:35.150 Now inside buttons set up. 00:15:35.150 --> 00:15:39.583 We want that when the class A button is pressed, we want to add 00:15:39.583 --> 00:15:47.050 the current frame from our video input with a label A to our classifier. 00:15:47.133 --> 00:15:48.333 So inside button set 00:15:48.333 --> 00:15:51.450 up, we want to access our button A. 00:15:51.483 --> 00:15:55.516 So let's say button A 00:15:55.600 --> 00:15:59.900 and as I said, we'll use the select function to access 00:15:59.966 --> 00:16:03.733 these variables that we have declared in our HTML file. 00:16:03.816 --> 00:16:07.550 So we use select and then we say hashtag 00:16:07.616 --> 00:16:11.083 button a. 00:16:11.166 --> 00:16:14.383 And when button is pressed, 00:16:14.466 --> 00:16:18.850 we want to add the current frame from our video 00:16:18.850 --> 00:16:23.233 feed with the label A to our classifier. 00:16:23.316 --> 00:16:26.316 So we'll do this by using the mouse 00:16:26.316 --> 00:16:29.983 pressed function. 00:16:30.066 --> 00:16:38.950 So button A dot mouse pressed, 00:16:39.033 --> 00:16:43.416 and we are going to declare our function for what we want to do. 00:16:43.416 --> 00:16:47.916 And the mouse is pressed for button a just directly here. 00:16:47.950 --> 00:16:53.916 So we just say function and we will 00:16:54.000 --> 00:16:59.100 call our model dot add image function, 00:16:59.100 --> 00:17:05.533 which is an ML5 function associated with these image classifiers. 00:17:05.616 --> 00:17:07.716 So we'll add image 00:17:07.716 --> 00:17:10.716 and we'll give it the label. A 00:17:10.800 --> 00:17:13.800 So that's as easy as that. 00:17:13.800 --> 00:17:18.150 And then we want to display our counter. 00:17:18.150 --> 00:17:22.333 So we want to update how many images we've collected of Class A. 00:17:22.416 --> 00:17:25.200 So again, we'll use select. 00:17:25.200 --> 00:17:27.600 We will be referring 00:17:27.600 --> 00:17:33.766 to our hashtag amount of 00:17:33.833 --> 00:17:40.450 class A, 00:17:40.533 --> 00:17:43.350 and that is in HTML. 00:17:43.350 --> 00:17:47.766 We want to display for that value. 00:17:47.850 --> 00:17:51.700 Um, our class a variable 00:17:51.700 --> 00:17:57.583 that we declared a top here, um, 00:17:57.666 --> 00:17:58.566 plus equal one. 00:17:58.566 --> 00:18:02.500 So we want to indent it by one and actually we should 00:18:02.583 --> 00:18:06.766 wrap this in its own 00:18:06.850 --> 00:18:09.066 parentheses like this 00:18:09.066 --> 00:18:13.966 so that this sum is what is displayed instead of the zero 00:18:14.050 --> 00:18:21.150 in our index. HTML. 00:18:21.233 --> 00:18:24.933 So now we can close off this button A 00:18:24.933 --> 00:18:27.333 mouse pressed, 00:18:27.416 --> 00:18:31.533 and we can add the same functionality 00:18:31.616 --> 00:18:36.250 basically identical for our function B. 00:18:36.333 --> 00:18:40.783 So I'm just going to copy this 00:18:40.850 --> 00:18:44.050 and paste it directly underneath 00:18:44.133 --> 00:18:47.500 and change from button A 00:18:47.583 --> 00:18:51.550 to button B, 00:18:51.633 --> 00:18:52.550 and we want to make sure 00:18:52.550 --> 00:18:55.550 that we are then adding the image 00:18:55.800 --> 00:19:00.283 with our ID B instead of aA 278 00:19:00,350 --> 00:19:02,933 and also displaying 00:19:02.933 --> 00:19:09.383 and updating the correct counter variable. 00:19:09.450 --> 00:19:10.350 So let's have a look 00:19:10.350 --> 00:19:14.150 at what our button setup looks like. 00:19:14.233 --> 00:19:19.700 Yeah, we have our image buttons to add data 00:19:19.783 --> 00:19:24.100 and we have label and confidence referring to the predictions 00:19:24.100 --> 00:19:27.683 from the model which we have yet to set up. 00:19:27.766 --> 00:19:30.550 And right now these buttons, when we press them, 00:19:30.550 --> 00:19:33.550 don't really do anything. 00:19:33.800 --> 00:19:35.483 So now we'll add a training 00:19:35.483 --> 00:19:40.083 and a predict button for our button setup function. 00:19:40.166 --> 00:19:41.833 So first, let's declare the 00:19:41.833 --> 00:19:45.966 buttons in our HTML file. 00:19:46.050 --> 00:19:49.050 So first, we want our training button. 00:19:49.166 --> 00:19:53.050 So again, we do button 00:19:53.133 --> 00:19:56.983 and we give it an ID 00:19:57.066 --> 00:20:03.916 should be trained 00:20:03.983 --> 00:20:08.500 and we give it a label 00:20:08.583 --> 00:20:11.166 again, which should just be train. 00:20:11.166 --> 00:20:18.066 And then we close this button definition. 00:20:18.150 --> 00:20:22.433 And while the model is training, we'll have access to 00:20:22.433 --> 00:20:24.333 what's called a loss value. 00:20:24.333 --> 00:20:30.400 And we can think of the loss value as a reverse measure of the model's accuracy. 00:20:30.416 --> 00:20:33.683 So the more accurate the model's predictions are during 00:20:33.716 --> 00:20:37.033 training, the lower our loss value will be. 00:20:37.116 --> 00:20:41.316 So let's access that value as well, because we want to be able to 00:20:41.400 --> 00:20:44.400 to print it as we are training. 00:20:44.516 --> 00:20:47.250 So we'll do a span, 00:20:47.250 --> 00:20:50.116 give it an ID 00:20:50.183 --> 00:20:56.650 loss and close that one 00:20:56.733 --> 00:21:03.516 and close our span 00:21:03.600 --> 00:21:05.666 like so. 00:21:05.666 --> 00:21:09.966 And then finally, we'll add our predict button. 00:21:10.050 --> 00:21:14.350 So again, we define our button 00:21:14.433 --> 00:21:17.766 with an ID 00:21:17.850 --> 00:21:21.183 equals 00:21:21.266 --> 00:21:24.266 let's call it button 00:21:24.300 --> 00:21:30.666 predict. 00:21:30.750 --> 00:21:33.033 And that's right. 00:21:33.033 --> 00:21:36.033 Start 00:21:36.033 --> 00:21:38.183 guessing 00:21:38.183 --> 00:21:39.983 for that button 00:21:39.983 --> 00:21:48.033 and then we will close it. 00:21:48.116 --> 00:21:50.250 And for good measure, let's add 00:21:50.250 --> 00:21:56.166 just copy this little line break, add a line break at the end. 00:21:56.250 --> 00:22:00.266 So let's go back to our sketch and deal 00:22:00.266 --> 00:22:04.566 with what to do when this training button is pressed. 00:22:04.650 --> 00:22:10.216 So in our button set up, we are going to 00:22:10.283 --> 00:22:14.433 add this train logic. 00:22:14.516 --> 00:22:18.150 So we use select to access this variable 00:22:18.150 --> 00:22:22.450 from our HTML file 00:22:22.533 --> 00:22:29.566 hash tag train. 00:22:29.633 --> 00:22:34.033 And then what we want to do when this 00:22:34.116 --> 00:22:36.283 button is pressed, 00:22:36.366 --> 00:22:37.283 we're going to do a 00:22:37.283 --> 00:22:41.916 similar thing as we did for 00:22:42.000 --> 00:22:43.766 adding our data. 00:22:43.766 --> 00:22:49.533 We're going to define this function just directly inside the call to mouse pressed. 00:22:49.616 --> 00:22:56.166 So we just write function 00:22:56.250 --> 00:22:57.866 and what we want it to do 00:22:57.866 --> 00:23:01.083 is we want to call model dot train, 00:23:01.166 --> 00:23:05.150 which initiates the training function from ML5, 00:23:05.150 --> 00:23:09.733 which is associated to this image classifier. 00:23:09.816 --> 00:23:14.966 And this function will take this loss 00:23:14.966 --> 00:23:20.833 value as its input. 00:23:20.916 --> 00:23:25.716 And what we want to do is access and display 00:23:25.716 --> 00:23:29.966 that lost value when it's available and when it is no longer available. 00:23:29.966 --> 00:23:36.483 We know that the model is done training so we can define a little let's see. 00:23:36.566 --> 00:23:43.366 So if there is a loss value, 00:23:43.433 --> 00:23:46.833 then we want to 00:23:46.916 --> 00:23:53.650 define a local variable loss 00:23:53.733 --> 00:23:56.016 that we can print. 00:23:56.016 --> 00:24:01.333 So again, we use select to access this 00:24:01.416 --> 00:24:04.833 00:24:04.916 --> 00:24:06.833 the HTML tag. 00:24:06.833 --> 00:24:10.833 So that's hash tag loss. 00:24:10.916 --> 00:24:12.933 And we want to replace that 00:24:12.933 --> 00:24:19.750 with the following HTML. 00:24:19.833 --> 00:24:22.833 We will replace it with 00:24:22.950 --> 00:24:25.433 this text loss 00:24:25.433 --> 00:24:30.783 and then using the dollar sign 00:24:30.866 --> 00:24:33.066 and curly brackets, 00:24:33.066 --> 00:24:37.866 we can refer to our local variable loss and print that. 00:24:37.950 --> 00:24:40.500 So notice that these 00:24:40.500 --> 00:24:44.166 these guys, these little quotation marks are different. 00:24:44.250 --> 00:24:48.300 They're not the double quotation marks that we use most of the time. 00:24:48.383 --> 00:24:52.150 They are these little lean back quotation marks, 00:24:52.216 --> 00:24:57.633 which makes it possible for us to refer to our variables, local variables 00:24:57.716 --> 00:25:01.166 with this dollar sign and the curly brackets 00:25:01.250 --> 00:25:02.933 to make sure that you use the right ones. 00:25:02.933 --> 00:25:06.916 It should be context highlight it in a way like this 00:25:06.916 --> 00:25:11.116 where you can see that loss is highlighted differently from the rest of the text. 00:25:11.116 --> 00:25:15.483 That's when you know that you've you've done it correctly. 00:25:15.566 --> 00:25:24.266 So you do like this. 00:25:24.350 --> 00:25:26.866 And as I mentioned, 00:25:26.866 --> 00:25:30.666 if we don't have a loss value, 00:25:30.750 --> 00:25:33.500 we can print something else. 00:25:33.500 --> 00:25:37.050 So we'll still use select 00:25:37.133 --> 00:25:43.050 and access this loss tag, 00:25:43.133 --> 00:25:46.416 but we will replace it with 00:25:46.500 --> 00:25:50.883 something else because now we no longer have a loss value. 00:25:50.966 --> 00:25:53.666 So let's just write 00:25:53.666 --> 00:25:58.900 done training 00:25:58.966 --> 00:26:06.716 and we can also print the final loss. 00:26:06.800 --> 00:26:10.300 So again, we're going to need to replace 00:26:10.366 --> 00:26:14.466 these quotation marks with the strange ones 00:26:14.550 --> 00:26:18.516 so that we can refer to our loss variable. 00:26:18.600 --> 00:26:20.783 Let's do this. 00:26:20.783 --> 00:26:24.900 And that's a dollar sign and 00:26:24.983 --> 00:26:31.600 curly brackets and loss. 00:26:31.666 --> 00:26:34.366 So now we can close out these functions. 00:26:34.366 --> 00:26:39.333 It might be a bit confusing to have these sort of nested function calls, 00:26:39.416 --> 00:26:43.550 but it also makes our code a little bit neater. 00:26:43.733 --> 00:26:47.200 And even though it can be hard to follow, 00:26:47.266 --> 00:26:50.383 it shortens down the amount of code that we need to need to write. 00:26:50.383 --> 00:26:58.916 So hopefully you think it's okay. 00:26:59.000 --> 00:27:01.850 And now let's set up our predict button. 00:27:01.850 --> 00:27:04.850 Now, this is luckily a lot simpler, 00:27:05.033 --> 00:27:08.033 so that's just a button predict 00:27:08.116 --> 00:27:13.200 equals and then we will select the button from our HTML. 00:27:13.283 --> 00:27:15.566 So hash tag I think we also just called it 00:27:15.566 --> 00:27:24.416 button predicts. 00:27:24.500 --> 00:27:29.250 And when this button is pressed 00:27:29.333 --> 00:27:31.166 so Mouse 00:27:31.166 --> 00:27:35.066 pressed is registered for this button. 00:27:35.150 --> 00:27:38.066 We want the model to start predicting. 00:27:38.066 --> 00:27:39.683 We'll start classifying. 00:27:39.683 --> 00:27:41.750 So let's declare a function. 00:27:41.750 --> 00:27:45.116 Classify. 00:27:45.200 --> 00:27:48.083 So just outside of our button 00:27:48.083 --> 00:27:52.266 setup function, we will declare a new function. 00:27:52.350 --> 00:27:57.666 Classify. 00:27:57.750 --> 00:27:58.883 Now here 00:27:58.883 --> 00:28:03.933 we can access our models, so model dot classify. 00:28:04.016 --> 00:28:07.900 This will call ML5´s classify function 00:28:07.966 --> 00:28:11.816 and in 00:28:11.816 --> 00:28:17.366 addition to calling this function, we need to give it a callback 00:28:17.366 --> 00:28:21.450 function the same way we did for model ready. 00:28:21.616 --> 00:28:25.400 We want a callback function that can handle what we do 00:28:25.583 --> 00:28:28.466 when the model has given some results. 00:28:28.466 --> 00:28:31.200 So let's define that one to say just 00:28:31.200 --> 00:28:33.950 we will call it got 00:28:33.950 --> 00:28:36.566 results. 00:28:36.566 --> 00:28:39.566 That's what I usually call them. 00:28:39.716 --> 00:28:51.616 Let's define that below. 00:28:51.700 --> 00:28:53.883 So inside got results. 00:28:53.883 --> 00:28:58.433 If you have taken part in our previous workshops, you know that 00:28:58.500 --> 00:29:01.916 we need to handle potential errors. 00:29:01.933 --> 00:29:03.283 At least that's good. 00:29:03.283 --> 00:29:05.400 Good policy to do that. 00:29:05.400 --> 00:29:08.466 So call results will return a potential error 00:29:08.466 --> 00:29:11.466 and some results. 00:29:11.533 --> 00:29:14.766 So the first thing we will do is to check 00:29:14.766 --> 00:29:17.766 if there is an error 00:29:17.766 --> 00:29:20.383 and if there is, 00:29:20.383 --> 00:29:25.750 we just want to log that error 00:29:25.833 --> 00:29:31.400 to our console window right down here. 00:29:31.483 --> 00:29:35.250 And if there is not an error, 00:29:35.283 --> 00:29:39.583 we can check the results. 00:29:39.666 --> 00:29:48.650 So if there are results 00:29:48.733 --> 00:29:54.016 for now, let's just print them. 00:29:54.100 --> 00:29:57.033 So console log, 00:29:57.033 --> 00:30:09.266 let's have a look at those results. 00:30:09.350 --> 00:30:10.766 So let's try this out. 00:30:10.766 --> 00:30:13.200 We can press play 00:30:13.200 --> 00:30:15.166 and we see everything is showing up. 00:30:15.166 --> 00:30:18.366 We now have our train and our predict button. 00:30:18.450 --> 00:30:21.733 So let's just add some images 00:30:21.816 --> 00:30:26.850 of some different things 00:30:26.933 --> 00:30:30.566 and press train. 00:30:30.650 --> 00:30:34.050 So now we see the loss is decreasing very quickly. 00:30:34.050 --> 00:30:37.616 And this is done training and it gives us the final loss. 00:30:37.800 --> 00:30:39.450 So let's start guessing. 00:30:39.450 --> 00:30:43.166 So when we press, start guessing or predict button, 00:30:43.250 --> 00:30:45.500 we will be calling classify, 00:30:45.500 --> 00:30:48.150 which gives us this call to got results. 00:30:48.150 --> 00:30:52.616 And we'll either get an error or we will be able to see what our results 00:30:52.616 --> 00:30:55.316 from the model might look like. 00:30:55.400 --> 00:30:55.750 So you press 00:30:55.750 --> 00:31:00.216 start getting and we did get something back, not an error. 00:31:00.300 --> 00:31:03.500 We get two objects back and the reason that we get two objects 00:31:03.500 --> 00:31:06.800 is because we are training a classifier 00:31:06.800 --> 00:31:10.566 with two labels, two possible classes. 00:31:10.650 --> 00:31:16.350 So if we trained a classifier with five classes, we would get five objects. 00:31:16.433 --> 00:31:19.500 And the way that these are structured, the results 00:31:19.733 --> 00:31:25.083 is that the first object I can move this so you can see it maybe a little easier. 00:31:25.166 --> 00:31:29.633 The first object will be the class that has received the highest confidence. 00:31:29.633 --> 00:31:31.366 So this is what the model is 00:31:31.366 --> 00:31:35.850 predicting is the correct class for a given input image. 00:31:35.933 --> 00:31:38.783 So if we look here, this would have been class B 00:31:38.783 --> 00:31:42.416 and it has around a 50%, 53% 00:31:42.500 --> 00:31:46.533 confidence, while the second is 46. 00:31:46.616 --> 00:31:50.283 So now we know sort of what this looks like. 00:31:50.366 --> 00:31:54.866 We can deal with it in a more reasonable way. 00:31:54.900 --> 00:31:57.500 So in got results. 00:31:57.500 --> 00:32:01.666 So if we have results now, we've seen them, we can comment this out. 00:32:01.666 --> 00:32:05.516 We don't need to look at that anymore 00:32:05.600 --> 00:32:12.216 and we can see 00:32:12.300 --> 00:32:18.100 add these curly brackets 00:32:18.166 --> 00:32:19.516 and inside this. 00:32:19.516 --> 00:32:24.700 So if there are results, we are actually just interested in this first objects. 00:32:24.766 --> 00:32:29.783 So we now know that result is basically a list of objects. 00:32:29.783 --> 00:32:34.133 So not only do we care about whether we have results, we need to know 00:32:34.133 --> 00:32:35.000 what is that? 00:32:35.000 --> 00:32:37.950 There's something in results zero. 00:32:37.950 --> 00:32:43.000 So the first element, which is the most 00:32:43.066 --> 00:32:44.566 likely class of the class 00:32:44.566 --> 00:32:48.600 that has been predicted to be correct. 00:32:48.683 --> 00:32:50.933 So we check both results and results 00:32:50.933 --> 00:32:53.933 zero and if we have both of them, 00:32:54.150 --> 00:32:58.800 then we can print our results. 00:32:58.800 --> 00:33:02.516 So now we're going to change these labels and confidence 00:33:02.516 --> 00:33:05.600 because that's what we now have have access to through these results. 00:33:05.600 --> 00:33:10.116 So let's say select 00:33:10.200 --> 00:33:12.450 are going to access 00:33:12.533 --> 00:33:18.700 results 00:33:18.766 --> 00:33:21.766 and we're going to replace what it now 00:33:21.766 --> 00:33:28.000 says, which is just label. 00:33:28.066 --> 00:33:29.150 And this we're actually 00:33:29.150 --> 00:33:32.150 replacing the dot, dot, dot, 00:33:32.183 --> 00:33:36.700 and we want to replace it with those 00:33:36.766 --> 00:33:39.383 results. 00:33:39.383 --> 00:33:44.583 Zero, which is the top prediction. 00:33:44.666 --> 00:33:48.200 And as we saw so we have results 00:33:48.200 --> 00:33:54.933 zero has this both label attribute and it has a confidence attribute. 00:33:55.016 --> 00:33:57.533 So that's what we want to access. 00:33:57.533 --> 00:34:00.500 So. results zero 00:34:00.500 --> 00:34:05.550 dot label 00:34:05.633 --> 00:34:13.350 should give us access to that. 00:34:13.416 --> 00:34:17.583 Let's see, 00:34:17.666 --> 00:34:19.066 do like this. 00:34:19.066 --> 00:34:20.550 So that's our results. 00:34:20.550 --> 00:34:26.166 And then we also want to select this confidence value. 00:34:26.250 --> 00:34:34.916 So that's hashtag confidence. 00:34:35.000 --> 00:34:38.650 And we want to replace its value 00:34:38.716 --> 00:34:42.816 with the confidence value that is in this object. 00:34:42.900 --> 00:34:46.150 So again, we want 00:34:46.216 --> 00:34:48.916 results, 00:34:48.916 --> 00:34:57.183 confidence, but results zero is what we want 00:34:57.266 --> 00:35:01.000 Dot Confidence. 00:35:01.066 --> 00:35:02.066 And as 00:35:02.066 --> 00:35:05.400 we can see here, this is a very long value. 00:35:05.400 --> 00:35:07.800 This has lots of lots of decimals. 00:35:07.800 --> 00:35:12.083 So think is good to minimize that. 00:35:12.316 --> 00:35:20.100 So we can do we can call a two fixed, 00:35:20.183 --> 00:35:23.516 not fixes fixed 00:35:23.600 --> 00:35:27.066 and let's say two decimals is enough. 00:35:27.150 --> 00:35:32.500 So two fixed and we can get them in percentage 00:35:32.566 --> 00:35:38.966 if we then times this by 100, 00:35:39.050 --> 00:35:42.700 that gives us 00:35:42.766 --> 00:35:47.050 the confidence in percentage. 00:35:47.116 --> 00:35:51.716 So let's save this and reset. 00:35:51.783 --> 00:35:54.066 And so the last thing that we want to do 00:35:54.066 --> 00:35:58.566 inside our got results function is that as long 00:35:58.566 --> 00:36:02.416 as there hasn't been an error, we want it to keep classifying. 00:36:02.416 --> 00:36:05.733 So let's call the classify function that we defined. 00:36:05.983 --> 00:36:07.966 Again. 00:36:07.966 --> 00:36:12.950 Now let's test this model out. 00:36:13.033 --> 00:36:18.733 Now I have this Lego flower 00:36:18.816 --> 00:36:20.166 and so 00:36:20.166 --> 00:36:23.433 let's try and train the model to recognize 00:36:23.433 --> 00:36:28.433 when I am holding the flower versus when I am not holding the flower. 00:36:28.516 --> 00:36:32.566 So let's start by adding some images for Class 00:36:32.566 --> 00:36:35.933 A, which will be our flower class 00:36:36.016 --> 00:36:39.466 just waving it around so that it and I'm just 00:36:39.466 --> 00:36:43.316 clicking away, adding images, 00:36:43.383 --> 00:36:45.066 and then 00:36:45.066 --> 00:36:47.266 I'll put my flower away. 00:36:47.266 --> 00:36:49.633 I'll just add some images 00:36:49.633 --> 00:36:53.183 of just me sitting here, maybe waving my hand, 00:36:53.266 --> 00:36:56.533 but there is no no flower. 00:36:56.616 --> 00:37:01.166 I'm going to try and add equal amounts of 00:37:01.233 --> 00:37:04.866 with and without flower classes so that the training is balanced 00:37:04.866 --> 00:37:10.033 and the model will have a chance to be more robust at recognizing 00:37:10.116 --> 00:37:11.983 the two different classes. 00:37:11.983 --> 00:37:14.233 So yeah, we've collected a bunch of data. 00:37:14.233 --> 00:37:17.066 Now we can click train. 00:37:17.133 --> 00:37:18.783 This will take hopefully a little bit longer. 00:37:18.783 --> 00:37:22.483 That's still very fast, even though we did add some more data 00:37:22.483 --> 00:37:26.333 this time. 00:37:26.416 --> 00:37:29.116 And so we can press start guessing. 00:37:29.116 --> 00:37:35.083 So now it's guessing with 100% confidence that I do not have a flower, but 00:37:35.166 --> 00:37:39.416 if I'm over here, then maybe I do have a flower. 00:37:39.483 --> 00:37:42.483 Let's see if I bring the flower 00:37:42.616 --> 00:37:43.933 with it. No. 00:37:43.933 --> 00:37:44.983 Yeah. 00:37:44.983 --> 00:37:49.883 It's very sure that it's seeing a flower this time. 00:37:49.966 --> 00:37:58.116 So we've now made a small flower detector. 00:37:58.200 --> 00:38:01.216 To make it more clear what our model is predicting. 00:38:01.433 --> 00:38:05.266 We can generate some visual feedback as well, so we can trigger 00:38:05.266 --> 00:38:09.850 an image to pop up on our canvas when a certain class is detected. 00:38:09.916 --> 00:38:13.766 Now I have these images of a robot face, so let's make this program 00:38:13.766 --> 00:38:16.883 a little more, at least for by having it show a happy face 00:38:16.883 --> 00:38:21.333 when a flower is recognized and a sad face when it's missing. 00:38:21.416 --> 00:38:24.416 So first, we'll add some images to display 00:38:24.450 --> 00:38:29.216 so we can do that by going into our file section here, sketch files 00:38:29.300 --> 00:38:35.066 and pressing the plus button and upload file. 00:38:35.150 --> 00:38:36.600 So I have a couple of 00:38:36.600 --> 00:38:41.100 files happy, neutral and sad robot, 00:38:41.216 --> 00:38:45.750 so I'm just going to drag them here and place them 00:38:45.833 --> 00:38:48.716 in my script. 00:38:48.716 --> 00:38:51.466 So there they are. 00:38:51.466 --> 00:38:53.300 Now we can see that they've popped up here 00:38:53.300 --> 00:38:58.283 with my sketch files. 00:38:58.366 --> 00:38:59.566 So now we can add the following 00:38:59.566 --> 00:39:03.116 to the top of our file here. 00:39:03.200 --> 00:39:08.850 We are going to declare some variables. 00:39:08.933 --> 00:39:11.683 So let's say 00:39:11.683 --> 00:39:14.683 our happy face 00:39:14.716 --> 00:39:19.716 and our sad face 00:39:19.783 --> 00:39:24.900 and our neutral face 00:39:24.983 --> 00:39:26.666 like this. 00:39:26.666 --> 00:39:31.283 And then we're going to define a preload function. 00:39:31.366 --> 00:39:34.666 So usually setup is the function 00:39:34.666 --> 00:39:39.150 that runs first in a P5 script, 00:39:39.233 --> 00:39:41.866 but preload actually runs 00:39:41.866 --> 00:39:44.816 before even before setup. 00:39:44.816 --> 00:39:47.816 So here we can sort of 00:39:47.816 --> 00:39:51.316 load the images before the program 00:39:51.550 --> 00:39:54.550 properly starts running and it doesn't slow things down. 00:39:54.550 --> 00:39:59.583 So in here we'll do image happy 00:39:59.666 --> 00:40:00.266 and we're going to 00:40:00.266 --> 00:40:04.050 use the function load image. 00:40:04.133 --> 00:40:06.566 We have to say what where it is. 00:40:06.566 --> 00:40:11.316 And since now it's just right here in our project folder, 00:40:11.383 --> 00:40:13.516 we can just refer to it by name. 00:40:13.516 --> 00:40:18.750 So that's happy dot JPEG 00:40:18.833 --> 00:40:20.233 and we're going to do the same 00:40:20.233 --> 00:40:23.633 for our sad face. 00:40:23.716 --> 00:40:26.000 So load image 00:40:26.000 --> 00:40:32.616 and this is sad dot JPEG 00:40:32.683 --> 00:40:44.766 And finally we are going to load our neutral 00:40:44.833 --> 00:40:47.716 load image 00:40:47.716 --> 00:40:53.066 and that's 00:40:53.150 --> 00:40:55.616 neutral dot 00:40:55.616 --> 00:41:00.750 jpeg. 00:41:00.833 --> 00:41:02.583 So now we want our canvas 00:41:02.583 --> 00:41:05.583 to show up in addition to our video input. 00:41:05.633 --> 00:41:08.633 So let's make a draw function. 00:41:08.883 --> 00:41:11.133 So the P5 draw function 00:41:11.133 --> 00:41:14.033 runs in a loop as long as the script is running. 00:41:14.033 --> 00:41:18.033 And initially we will just display a neutral face. 00:41:18.116 --> 00:41:21.416 But when the classifiers detected the flower or the lack of flower, 00:41:21.600 --> 00:41:25.383 we want it to shift the image to either happy or sad. 00:41:25.466 --> 00:41:28.316 So let's begin by loading 00:41:28.316 --> 00:41:38.733 just the neutral image. 00:41:38.816 --> 00:41:40.916 And then if 00:41:40.916 --> 00:41:44.166 we detect happy, 00:41:44.250 --> 00:41:48.900 we want to display 00:41:48.983 --> 00:41:50.433 all or happy image. 00:41:50.433 --> 00:41:53.916 So we're going to use the image function for this. 00:41:54.000 --> 00:41:58.633 So first off, the first argument should be our 00:41:58.700 --> 00:42:00.150 image file. 00:42:00.150 --> 00:42:07.350 She's our happy image, and then we have to position it on our canvas. 00:42:07.433 --> 00:42:10.283 So we want it to 00:42:10.283 --> 00:42:14.850 start all the way in our top left corner 00:42:14.933 --> 00:42:20.050 and we want its width and height to be the same size as our canvas. 00:42:20.066 --> 00:42:23.033 So canvas 00:42:23.033 --> 00:42:25.883 width 00:42:25.883 --> 00:42:32.166 and canvas height. 00:42:32.250 --> 00:42:33.500 So that's what should happen 00:42:33.500 --> 00:42:42.150 if the classifier detects our flower and if it doesn't. 00:42:42.233 --> 00:42:46.716 So if sad, 00:42:46.800 --> 00:42:51.550 then we want the sad image to show so we can just copy. 00:42:51.633 --> 00:42:55.750 Really, this whole this whole if statement. 00:42:55.833 --> 00:43:02.666 So if sad, we want the sad image to show up. 00:43:02.750 --> 00:43:03.850 Now, to make this work, we 00:43:03.850 --> 00:43:09.683 want to add some logic to our function for handling the classifier results. 00:43:09.766 --> 00:43:15.300 So let's make to these two boolean values happy and sad 00:43:15.366 --> 00:43:19.250 so we'll make them global. 00:43:19.333 --> 00:43:25.050 So let happy and sad initially be false. 00:43:25.116 --> 00:43:27.800 So none of them are triggered initially. 00:43:27.800 --> 00:43:31.500 And then inside our got results 00:43:31.516 --> 00:43:32.716 function. 00:43:32.716 --> 00:43:35.383 Let's see here. 00:43:35.383 --> 00:43:39.666 This is where we want to check exactly what label is being detected. 00:43:39.666 --> 00:43:42.850 So instead of just printing the label, this small text, 00:43:42.850 --> 00:43:45.850 we want to actually deal with 00:43:46.100 --> 00:43:50.100 different circumstances for different labels being predicted. 00:43:50.166 --> 00:43:53.166 So if 00:43:53.266 --> 00:43:57.166 results 00:43:57.250 --> 00:44:03.833 zero, if it's label is equal to 00:44:03.916 --> 00:44:06.916 is equal to a 00:44:07.100 --> 00:44:09.550 should 00:44:09.550 --> 00:44:11.050 get the context right 00:44:11.050 --> 00:44:14.250 by wrapping this and then curly brackets. 00:44:14.316 --> 00:44:17.650 So if the results label is equal to a 00:44:17.683 --> 00:44:21.900 we want to set happy to true 00:44:21.966 --> 00:44:23.383 and sad 00:44:23.383 --> 00:44:26.783 to false in case it's been changed 00:44:26.866 --> 00:44:30.650 in a previous frame 00:44:30.733 --> 00:44:34.050 and we want to do basically the opposite. 00:44:34.116 --> 00:44:36.466 So we're just going to copy and paste again. 00:44:36.466 --> 00:44:39.466 So if the label equals B, 00:44:39.650 --> 00:44:43.916 then we want to set happy to false 00:44:44.000 --> 00:44:48.616 and sad to true. 00:44:48.700 --> 00:44:49.616 Now, I notice that 00:44:49.616 --> 00:44:53.866 this has been highlighted and I can see that I have misspelled this. 00:44:53.950 --> 00:44:57.316 So results, it's. 00:44:57.383 --> 00:45:00.183 Yeah, and the same above 00:45:00.183 --> 00:45:03.650 results 00:45:03.716 --> 00:45:09.766 that looks more reasonable. 00:45:09.850 --> 00:45:12.916 And so the last thing that we should do 00:45:12.916 --> 00:45:15.916 is that we no longer have to hide our canvas. 00:45:16.000 --> 00:45:20.450 So that's remove this line of code 00:45:20.533 --> 00:45:23.450 and we can save our sketch. 00:45:23.450 --> 00:45:23.783 Okay. 00:45:23.783 --> 00:45:27.216 So let's retrain our model 00:45:27.300 --> 00:45:29.250 for flower detection 00:45:29.250 --> 00:45:32.100 and see what it looks like. 00:45:32.100 --> 00:45:32.433 Okay. 00:45:32.433 --> 00:45:35.216 So I'm going to make the code a little smaller. 00:45:35.216 --> 00:45:37.983 Actually, I can just hide this so that they 00:45:37.983 --> 00:45:45.483 are side by side and we can see everything that's going on. 00:45:45.566 --> 00:45:48.300 So let's start by 00:45:48.300 --> 00:45:51.783 adding some images for Class A, 00:45:51.866 --> 00:45:59.250 which should be the flower class, 00:45:59.333 --> 00:46:01.133 just adding those here. 00:46:01.133 --> 00:46:05.683 So then I'll do the same 00:46:05.750 --> 00:46:07.233 without the flower. 00:46:07.233 --> 00:46:12.783 Just my empty hand 00:46:12.866 --> 00:46:15.116 like this. 00:46:15.116 --> 00:46:16.850 And let's start training. 00:46:16.850 --> 00:46:19.850 So now we see that 00:46:20.100 --> 00:46:23.916 he is neutral, the little robot. 00:46:24.000 --> 00:46:28.383 And when we start guessing becomes very sad 00:46:28.466 --> 00:46:30.950 because there's no flower 00:46:30.950 --> 00:46:34.716 and he becomes happy when the flower is present. 00:46:34.800 --> 00:46:37.533 So our flower detection is working. 00:46:37.533 --> 00:46:42.450 We are pleasing this little robot face. 00:46:42.533 --> 00:46:44.516 I'm sorry. 00:46:44.516 --> 00:46:47.150 Okay, let's 00:46:47.150 --> 00:46:50.100 reset the application 00:46:50.100 --> 00:46:54.633 so it can become a little bit annoying to have to keep training 00:46:54.633 --> 00:46:58.850 and retraining our model every time that we reload our program 00:46:58.916 --> 00:47:01.383 or change something in our code. 00:47:01.383 --> 00:47:05.016 So to wrap things up, I thought we would implement functions 00:47:05.016 --> 00:47:08.950 for saving and loading our trained model. 00:47:09.033 --> 00:47:17.950 So again, we're going to add some new buttons to our HTML file. 00:47:18.033 --> 00:47:22.700 So let's find our HTML file, indexe dot HTML, 00:47:22.766 --> 00:47:28.000 and we are going to just under or 00:47:28.083 --> 00:47:31.666 let's put it at the very bottom under 00:47:31.750 --> 00:47:37.900 our predictions, we're going to add a button 00:47:37.983 --> 00:47:47.083 with the ID save 00:47:47.166 --> 00:47:49.233 and then we are going to give 00:47:49.233 --> 00:47:55.266 that some text save, save model. 00:47:55.350 --> 00:47:57.633 And then we have to close up 00:47:57.633 --> 00:48:00.966 all button definition 00:48:01.050 --> 00:48:02.433 like so. 00:48:02.433 --> 00:48:05.433 And we are going to add 00:48:05.500 --> 00:48:07.866 just some a little line 00:48:07.866 --> 00:48:11.266 break for readability. 00:48:11.333 --> 00:48:14.333 So that's our save button. 00:48:14.550 --> 00:48:17.616 And then we also want to add a button 00:48:17.616 --> 00:48:21.283 to load up a previously trained model. 00:48:21.366 --> 00:48:24.816 So the way that we do this is using 00:48:24.816 --> 00:48:28.033 something called a label. 00:48:28.116 --> 00:48:31.566 So the label 00:48:31.650 --> 00:48:34.716 and then we specify that it's 00:48:34.716 --> 00:48:38.950 for this 00:48:39.033 --> 00:48:41.050 for our 00:48:41.050 --> 00:48:44.316 avatar 00:48:44.400 --> 00:48:54.666 called load model, 00:48:54.750 --> 00:49:01.300 and then close the label definition. 00:49:01.383 --> 00:49:06.633 00:49:06.850 --> 00:49:11.283 And the input that we can expect from this load 00:49:11.316 --> 00:49:15.550 model button being pressed also needs to be defined here. 00:49:15.633 --> 00:49:18.633 So we do that as input 00:49:18.633 --> 00:49:21.450 and we specify it's type 00:49:21.450 --> 00:49:25.150 that this is a file 00:49:25.233 --> 00:49:27.850 and its ID 00:49:27.850 --> 00:49:31.916 should be equal to load. 00:49:31.916 --> 00:49:37.000 So this is what we are going to be referring to in our sketch. 00:49:37.083 --> 00:49:39.150 So in our P5 script. 00:49:39.150 --> 00:49:41.766 And we can also here specify 00:49:41.766 --> 00:49:47.066 that we want to be able to load multiple files. 00:49:47.150 --> 00:49:55.483 So we just write multiple. 00:49:55.566 --> 00:49:58.233 And so this little line 00:49:58.233 --> 00:50:04.166 should allow us to load multiple files and we'll have a little bit of a look 00:50:04.383 --> 00:50:12.200 at just what this, this sort of looks like. 00:50:12.283 --> 00:50:14.283 But before that, let's 00:50:14.283 --> 00:50:19.033 just see what our layout looks like 00:50:19.116 --> 00:50:21.583 and start the program. 00:50:21.666 --> 00:50:21.850 Okay. 00:50:21.850 --> 00:50:25.850 So it's a little bit jumbled. 00:50:25.933 --> 00:50:27.250 So I think what we will do 00:50:27.250 --> 00:50:30.866 is that we will use this p, 00:50:30.933 --> 00:50:35.083 which is basically just defining a 00:50:35.166 --> 00:50:37.716 HTML paragraph. 00:50:37.716 --> 00:50:41.600 Oh, not capital P, just regular p, 00:50:41.683 --> 00:50:43.833 So we'll start a paragraph 00:50:43.833 --> 00:50:48.883 before the save and load button 00:50:48.966 --> 00:50:51.583 and then we end it underneath. 00:50:51.583 --> 00:50:56.333 This should hopefully make it look a little bit cleaner. 00:50:56.416 --> 00:50:58.333 There we go. 00:50:58.333 --> 00:51:02.166 Now we can see that we have our save model button 00:51:02.166 --> 00:51:06.200 and our load model button, 00:51:06.283 --> 00:51:08.350 So let's reset and let's 00:51:08.350 --> 00:51:13.050 go back to our sketch and add the logic 00:51:13.050 --> 00:51:17.450 for these buttons into our button setup function. 00:51:17.516 --> 00:51:22.366 So here we are, 00:51:22.450 --> 00:51:25.116 we last edited our predict button. 00:51:25.116 --> 00:51:32.900 And so now let's make a save button. 00:51:32.966 --> 00:51:37.400 And we do that by selecting 00:51:37.466 --> 00:51:40.466 Our hashtag, 00:51:40.616 --> 00:51:45.466 save ID from our HTML file 00:51:45.550 --> 00:51:48.916 and we need to handle how 00:51:49.000 --> 00:51:53.833 this button should react when we press. 00:51:53.916 --> 00:51:59.300 So let's say mouse pressed 00:51:59.366 --> 00:52:01.750 What should we do when the mouse is pressed? 00:52:01.750 --> 00:52:03.716 Well, 00:52:03.716 --> 00:52:06.450 luckily there is a predefined 00:52:06.450 --> 00:52:10.300 function in ML5 to do with this, 00:52:10.383 --> 00:52:13.083 so let's do this 00:52:13.083 --> 00:52:16.783 compressed version of defining a function. 00:52:16.866 --> 00:52:20.250 So we'll just do it right here because there's not much we need to write. 00:52:20.250 --> 00:52:24.350 We seem to write Model dot save. 00:52:24.416 --> 00:52:28.000 Now what this does 00:52:28.083 --> 00:52:30.633 is the image classifier 00:52:30.633 --> 00:52:34.350 will create two files for us 00:52:34.500 --> 00:52:39.650 a model dot Json file and a model weights dot bin file 00:52:39.716 --> 00:52:45.283 and the first to the model dot JSON file that describes the structure of the model 00:52:45.366 --> 00:52:51.200 and the second file contains the weights that have resulted from our training. 00:52:51.266 --> 00:52:57.333 So make sure that you allow the browser to actually download 00:52:57.333 --> 00:53:03.500 multiple files from the P5 editor when you are prompted. 00:53:03.566 --> 00:53:06.600 Now the load button will open a file browser 00:53:06.600 --> 00:53:11.116 and will allow us to load the two files that we have created. 00:53:11.200 --> 00:53:15.216 So will also want to change the model status 00:53:15.216 --> 00:53:19.600 text to reflect that we have now loaded a custom model. 00:53:19.683 --> 00:53:25.000 So let's create our load button. 00:53:25.083 --> 00:53:28.400 So let's say load 00:53:28.466 --> 00:53:32.516 button equals select 00:53:32.600 --> 00:53:39.766 and we call this one load. 00:53:39.850 --> 00:53:41.850 And so what we want to do is we want to detect 00:53:41.850 --> 00:53:46.666 when this button has changed so we can do 00:53:46.750 --> 00:53:49.683 load button, 00:53:49.683 --> 00:53:54.050 dot changed, 00:53:54.116 --> 00:53:56.166 and we can in here 00:53:56.166 --> 00:54:01.750 define this function again in this compressed manner. 00:54:01.833 --> 00:54:03.633 And we're going to be using 00:54:03.633 --> 00:54:07.966 the built in model dot load 00:54:08.050 --> 00:54:08.966 from ML5. 00:54:08.966 --> 00:54:11.966 So model dot load 00:54:12.116 --> 00:54:17.366 and what we want to send to this 00:54:17.366 --> 00:54:21.700 ML5 function is basically the elected files. 00:54:21.700 --> 00:54:26.016 So the way we refer to that is we refer to our load button, 00:54:26.100 --> 00:54:30.550 load button dot 00:54:30.633 --> 00:54:35.316 elt dot files 00:54:35.400 --> 00:54:37.750 and then the load function 00:54:37.750 --> 00:54:43.650 also wants us to supply it with a callback function 00:54:43.650 --> 00:54:47.416 the same way we've done for got results and model ready. 00:54:47.500 --> 00:54:52.350 So here we will just send it a small function 00:54:52.350 --> 00:54:57.966 that just changes the status of our HTML to show 00:54:57.966 --> 00:55:01.933 custom model is loaded instead of just model ready. 00:55:02.016 --> 00:55:07.716 So select our 00:55:07.800 --> 00:55:14.766 model status variable 00:55:14.850 --> 00:55:18.966 and we want to change that to 00:55:19.050 --> 00:55:21.900 custom model loaded. 00:55:21.900 --> 00:55:27.683 And that way we can sort of easily see what model we are currently working with. 00:55:26.366 --> 00:55:30.833 currently working with. 00:55:30.900 --> 00:55:33.100 Okay, so let's train this model 00:55:33.100 --> 00:55:38.116 a final time and save it. 00:55:38.200 --> 00:55:41.933 Let's hide this again so we can see a little easier. 00:55:42.000 --> 00:55:44.733 Bring out my flower 00:55:44.733 --> 00:55:53.750 and add some images of it 00:55:53.833 --> 00:55:56.466 around 30 00:55:56.466 --> 00:56:03.766 and add some images without 00:56:03.850 --> 00:56:05.433 around the same amount. 00:56:05.433 --> 00:56:11.983 And then we can start training 00:56:12.066 --> 00:56:13.116 and start guessing. 00:56:13.116 --> 00:56:18.850 So sad without right and happy with. 00:56:18.933 --> 00:56:22.300 So when we're happy with our training 00:56:22.300 --> 00:56:27.500 and the accuracy of the model, we can do save. 00:56:27.583 --> 00:56:32.350 And so now it has been added 00:56:32.433 --> 00:56:34.000 our two files. 00:56:34.000 --> 00:56:37.283 Now we can choose them 00:56:37.350 --> 00:56:42.400 like so 00:56:42.483 --> 00:56:45.216 and we can start guessing. 00:56:45.216 --> 00:56:47.683 So now we see that it says two files. 00:56:47.683 --> 00:56:49.050 We say that it's working. 00:56:49.050 --> 00:56:51.883 And just to be doubly sure that we are 00:56:51.883 --> 00:56:54.883 loading the correct file, 00:56:55.083 --> 00:56:59.650 we can just restart the whole thing. 00:56:59.733 --> 00:57:03.100 And this time we won't add any images, we won't train. 00:57:03.183 --> 00:57:06.850 We'll just choose these model files, 00:57:06.933 --> 00:57:10.933 Model dot json and model weights dot bin, 00:57:11.016 --> 00:57:14.366 and then we can press start guessing and again. 00:57:14.383 --> 00:57:17.383 Yeah, he said without the flower 00:57:17.466 --> 00:57:24.550 and we can make him happy. 00:57:24.633 --> 00:57:29.900 So that's it for our 00:57:29.983 --> 00:57:32.550 image classification project. 00:57:32.550 --> 00:57:35.950 So you can take this further, if you like, as usual, 00:57:35.950 --> 00:57:40.633 by checking out the links on our website under the header going further. 00:57:40.716 --> 00:57:42.950 And yeah, thanks so much for joining.