WEBVTT 00:00.000 --> 00:05.000 [Automatic captions by Autotekst using OpenAI Whisper V3. May contain recognition errors.] 00:08.132 --> 00:17.838 Okay, so a little bit about sonification. What is sonification? Sonification is basically using data to make sound or more technically 00:19.619 --> 00:29.764 transforming data relationships into perceived auditory relationships. There are many motivations why you would do this, but I 00:29.824 --> 00:35.305 always try to say that the motivation is somewhat analogous to visualization. 00:37.566 --> 00:49.112 We developed this very efficient and sophisticated pattern recognition machine in our heads. and through our senses we are able to 00:49.192 --> 00:58.900 notice very subtle events, very subtle patterns and so both visualization and sonification are basically for leveraging this 00:59.021 --> 01:11.206 pattern recognition machine So I have this somewhat complicated graph that shows the potential problems and also the potential steps 01:11.827 --> 01:21.308 involved in either data visualization or data sonification or the two of them together like an audiovisual display. you know there are 01:21.588 --> 01:31.253 continuous physical phenomena in the world that we are that we learned to understand and observe or understand by observation so there are 01:31.293 --> 01:45.001 these very long arrows to our senses either eyes or ears and then through our senses we as we directly observe the world we make sense of it but when 01:45.041 --> 01:57.051 we use visualization or sonification then actually there are these several steps and all of them are in some way lossy transformations to 01:57.071 --> 02:10.502 these patterns until we actually again get to look at or listen to things so we can of course sample this reality in some way we can turn it into 02:10.562 --> 02:20.585 digital data There is a huge problem whether how these digital representations are already relevant to these things, as many of you 02:20.625 --> 02:32.227 know. And then we have to somehow create displays out of this digital data, either visually or orally. And then we actually get to look at 02:32.267 --> 02:41.970 those or listen to those and try to understand something about this real world process. Pixisonics, this toolbox that I'm working on, is 02:42.890 --> 02:54.297 somewhere in the overlap of this problem. So I'm trying to use particularly image data and then trying to create an interactive 02:54.497 --> 03:05.700 audio-visual display that helps you perhaps understand something what the image data is attempting to capture. and more technically or 03:05.720 --> 03:16.587 more closely what we are going to do today is a paradigm in sonification called parameter mapping sonification this is by far the most used 03:17.488 --> 03:28.490 technique in sonification and you're maybe already familiar with it so You take some dataset, some data of features that you care about, and 03:28.530 --> 03:39.373 then you take those features and map them into input parameters of a synth, something that makes sound. And then you synthesize those input 03:39.393 --> 03:49.476 parameters into a sound wave, and then you listen to that and try to infer something about the data relationships or distributions in your 03:49.536 --> 04:02.064 dataset here. so there are other ways to do sonification and also this might have some inherent problems but like the most common examples of 04:02.144 --> 04:09.650 sonification in the everyday world are actually all using or pretty much all of them are using this technique so if you think about like metal 04:09.690 --> 04:19.277 detectors or Geiger counters or heart rate monitors or like this parking assistant in your car when you're backing up for example 04:22.099 --> 04:33.306 all of them are using this paradigm and typically parameter mapping sonification can be very efficient when you are doing a sort of 04:33.446 --> 04:42.955 iterative design. It's very quick to prototype. So that's why I thought it should be sort of the main theme of this toolbox. This is what we will 04:43.716 --> 04:44.456 deal with today. 04:47.936 --> 04:58.123 I wanted to get this question out of the way relatively early on because when I started this PhD this was one of my main questions like why should 04:58.163 --> 05:11.112 we sonify what we already see so I see image sonification as a special sort of subfield in sonification because the data that we turn into 05:11.172 --> 05:24.362 sound is already in a representation that we can make sense of with our sensory apparatus. So why bother? Why shouldn't we just look at them? 05:25.603 --> 05:37.293 And the first and immediate to second answer is that, well, we have developed different sensory modalities for a reason and they have 05:37.333 --> 05:46.579 specialized differently so in other words there are patterns that are easier to see and maybe there are other kinds of patterns that are easier 05:46.639 --> 05:57.929 to hear the most simplest example is anything temporal and anything that has a high temporal resolution and in some ways I think of 05:58.830 --> 06:10.383 frequencies and pitch as a derivative of this temporal resolution as well so anything that is highly temporal is probably easier to hear than 06:10.423 --> 06:23.994 to see But okay, that's one thing. But what I'm really interested in, and I think that was also a main influence of conceiving what this toolbox 06:24.034 --> 06:35.466 should be about, is that While there are things easier to see or easier to hear, at the same time in our brain there are very specific circuits that 06:35.547 --> 06:45.077 are specializing in integrating these different sensory modality signals into an integrated audio-visual experience. What is 06:45.157 --> 06:57.665 interesting is that in newer sonification research people start to pitch that perhaps some audiovisual redundancy when designing 06:57.745 --> 07:08.052 audiovisual displays is actually better and they typically perform better with users than if you compare it to a system where, let's say, 07:08.072 --> 07:17.899 there's one type of data that you display visually and some kind of orthogonal data that you display orally. That also exists, and there 07:17.999 --> 07:28.188 are some, I mean, there are many studies that work like that, but newer studies seem to suggest that if you actually design some redundancy, 07:28.228 --> 07:40.593 people sort of catch it earlier, because in our heads, I think, This highly correlates with how we learned to understand the world, in my 07:40.613 --> 07:54.136 opinion. But okay, enough with that. What is Pixasonics about? So these are some sort of main aspirations that I have and had when I started 07:54.176 --> 08:05.098 designing this toolbox. So first of all, it is a library, not a web application. although it is kind of a crossover between a web 08:05.138 --> 08:15.362 application like a visual graphical interface and an actual library I wanted to do this so that it's kind of easy to get into you don't have to do a 08:15.422 --> 08:24.845 lot of code you just hit some lines and then suddenly you have this nice visual interface and you can do stuff with it so you can kind of get going 08:24.925 --> 08:38.315 pretty quickly but if you want to go deep there are ways to let you go deep Another very important part is that it's interactive. One thing is that 08:39.216 --> 08:50.488 audiovisual redundancy or maybe correlation, so some relationship between what you see and what you hear helps you to get a deeper model, to 08:50.588 --> 09:01.255 form a deeper understanding of what you're dealing with but another thing is interaction so if you are the one who controls sort of the audio 09:01.315 --> 09:15.344 and the visual patterns that reach you then there is this other loop that helps you in a way explore the data in a more natural way perhaps to to your 09:15.384 --> 09:27.270 cognition so i really really wanted to keep this very real time very interactive as much as Jupyter Notebook will let me. So I hope you will 09:27.310 --> 09:40.656 enjoy that part. I think that's mostly the fun part about this. And yeah, as with maybe any library, I aspire to make it generic, so not so many 09:40.736 --> 09:48.980 ready-made results, but more about giving you tools to build your own thing. and hopefully flexible because one thing about image 09:49.000 --> 09:56.824 sonification that I learned throughout this research is that it's always different and everybody has different kinds of data different 09:56.884 --> 10:08.610 kinds of workflows so it's actually very hard to make this one thing that will satisfy everyone and maybe it will never happen but I tried to at 10:08.690 --> 10:21.325 least give you some kind of break out doors so that you can integrate this with other typical scientific computing libraries in Python. and then 10:21.345 --> 10:31.992 the final big disclaimer is that it's very very very alpha I think the entire code base is a little bit more than a month old there has been 10:32.032 --> 10:40.318 already a workshop on it which was successful and since then there were a ton of optimizations so I think it should be fine but you know these are 10:40.358 --> 10:56.806 famous last words so yeah just be gentle and be patient and I'm very open for feedback so With all that said, let's get into it. I showed you these 10:56.906 --> 11:07.130 commands earlier and I think hopefully you got the email about them as well. But just a quick explanation or recap again what they do. First of 11:07.250 --> 11:19.958 all, you don't really have to have a virtual environment for this. um i just decided to very strongly suggest it so that i avoid problems of like 11:19.978 --> 11:27.763 spurious installation errors because pip is not super strong in like interlacing many different requirements from different 11:27.863 --> 11:35.608 dependencies i mean it doesn't have a lot of dependencies but still it's in a way cleaner to do it in a virtual environment so this uses Anaconda or 11:35.688 --> 11:48.636 Miniconda to create a virtual environment called C2HO and please give me Python 3.12 and please give me pip installed with it because then when 11:48.676 --> 12:02.746 we activate this environment it should be just pip install Pixisonics Pixisonics is the name of the library It's a wordplay of pixel, as in 12:03.306 --> 12:16.459 ImagePixel and Sonix. I usually like to have this nocached here, because I observed that Anaconda can sometimes use cached libraries 12:16.560 --> 12:24.926 from other environments, which is kind of weird. honestly you should probably get away with just saying they can stop xs max but no cache tier 12:24.986 --> 12:35.991 is like another little safety man and then once you have that installed it usually takes a minute or two Then there is this terminal command that 12:36.071 --> 12:47.838 I set up so you can just say Pixasonics-notebook and then it will launch the actual Jupyter notebook. So I'll try to demonstrate this now. So 12:47.958 --> 12:59.357 here is my terminal. Yeah, my environment is named Notom, but that shouldn't confuse you. And I have everything installed, so I can just 12:59.397 --> 13:15.414 say Pixisonics notebook. and then it also tells you where the notebook is that it launches and then you get to here so hopefully this works for 13:15.434 --> 13:28.803 you and then there is this very long kind of like a storyline that i build up and yeah as you see as we get to the advanced examples there is less and 13:28.843 --> 13:39.572 less blah blah but i'll get to that later So what we will get through today is pretty much this notebook. So the first part is more the high-level 13:39.592 --> 13:53.383 gist and sort of step-by-step gentle introduction. And then I hope that we will get all these advanced use cases, which will be a bit more codey 13:53.423 --> 14:06.118 and tech savvy. But yeah, that's the menu. So let's get into it. I like to just jump in and actually just show you a little bit of code right away. 14:06.499 --> 14:20.609 Yeah, let's start with this first cell. If I just evaluate it. it will do this. It will create this little interface with like this example image 14:20.709 --> 14:33.831 loaded in. So I very quickly, you don't have to understand this yet. You can just like one third or half understand it. So here I import from the 14:33.891 --> 14:43.394 core, the app and the mapper classes. Here from the features, I import the mean channel value feature. Then from the synths I import the 14:43.434 --> 14:57.822 teramin. Then I create an app, which by default has a canvas of 500x500 pixels. Then I load an image file. This test.jpg is just a UV map, so it's 14:57.882 --> 15:16.277 just, you know, on the three color channels we are ramping from 0 to 255 in these axes, just a test image. Then we create a feature, and we will learn 15:16.317 --> 15:27.667 more about this later, and we will say we will filter on the red channel and name it MeanRed. And then we say, hey app, attach this MeanRed 15:27.928 --> 15:41.393 feature to yourself. so that then in this features tab we will see the mean red features UI card. The aesthetics is very 90s because it's a 15:41.453 --> 15:52.596 Jupyter Notebook using the ipywidgets library and you know it's not like some very fancy CSS framework or something so it's gonna be very 15:52.656 --> 16:03.382 simple looking. So we create this and we attach it, good. So for this parameter mapping, we need a feature, we need a synth, and then we have to 16:03.422 --> 16:14.048 create a mapping between them. And this is basically the gist of the entire library. So you have features, you have synths, and then you have 16:14.128 --> 16:24.435 mappers that you put between features and synths. And then actually you put all three of these into an app, and that's your kind of whole package. 16:25.376 --> 16:34.508 So now we are creating a theremin from this theremin, and we name it MySign. This is just going to be an extremely simple sine wave generator 16:34.648 --> 16:45.222 with some additional controls. And then we attach, again, app.attach.theremin. Cool. I'm trying to scroll there. So in the 16:45.262 --> 16:56.676 synths, we have this side wave that has some sliders for their parameters. Again, extremely simple. And if I can come back here. The 16:56.736 --> 17:07.144 final piece, we have the feature, we have the synth. Now we have to have a mapper. So this red2frac is going to be a mapper that is saying take the 17:07.204 --> 17:09.045 mean red, which is our 17:13.086 --> 17:24.278 feature, and then map it to the frequency or parameter of the theremin. and then give it an exponent of 2 and name it red2frac. And then again ask 17:24.318 --> 17:32.968 the app to please attach this mapper to yourself. Sorry. 17:36.003 --> 17:50.420 So in this mappers pane now we have this and it says from mean red to my sign frequency. Good. That was a lot. Now let's zoom out a little bit and go 17:50.480 --> 17:53.203 back to how this kind of conceptually looks like. 17:57.732 --> 18:07.485 This is my attempt to kind of summarize the entire library in one shot. It's a little bit complex, but bear with me, we can walk through the 18:07.525 --> 18:11.811 different parts. I think it basically tells everything. 18:14.555 --> 18:27.050 if not so simply so so basically we have four modules that we use what we have just seen we create an app this says app here that does some things we 18:27.110 --> 18:37.558 have a feature which also does some things we have a synth that also does some things and a mapper and so I don't know if it shows but there are some 18:37.578 --> 18:47.285 of these kind of ghost copies behind all of these cards that's meant to show that all of this is modular and you can have as many of any of these as 18:47.325 --> 18:58.749 you want but okay let's see now what happens and how are these things really interconnected so when you create an app you see in this little UI 18:58.789 --> 19:09.792 this was kind of that little canvas and next to that these settings things so this is what it has and then what the app is responsible for is 19:10.132 --> 19:22.560 containing an image giving you a way to probe a little section of that image and to connect to a global real-time audiograph and sort of create 19:22.580 --> 19:34.492 this pipeline for other things that you can kind of fill in as you want. You can have several apps with different bespoke setups but you can also 19:34.572 --> 19:46.661 just have one mega app with all the things you want in there. So yeah, the important thing is that this connects to a global audiograph there. 19:47.401 --> 19:57.365 Now, what we do here is that we have an image. Don't worry about this 4D yet, but just think about it as an image. And then we have this little 19:57.505 --> 20:07.628 rectangle, this little probe that lets us take out part of the image, interactively so, in real time. And then basically what happens is that 20:08.009 --> 20:16.491 all the... You've seen that now we attached, like explicitly asked the app, hey, attach this feature, attach this synth, attach this mapper. 20:16.871 --> 20:25.838 So once we have this attached in the app's pipeline, then the app will actually send this slice of the image to all the features which will 20:26.238 --> 20:36.288 extract some kind of feature out of it and then the mappers will read from these feature buffers and then map it to parameter buffers for our 20:36.348 --> 20:45.733 synths and so these parameter buffers will get synthesized into sound And since we attached the synth to the app, it will connect to this audio 20:45.793 --> 20:55.355 graph so that we can hear everything. So again, this is a bit too much, so don't worry about this. This is kind of just a high level gist. So we have 20:55.375 --> 21:03.777 apps where there are images and a probe, and then we kind of attach all these features, synths and mappers to an app. And all of this can be 21:03.937 --> 21:12.980 multiplied and combined in various ways. In a way this is all what we will go through today. 21:13.940 --> 21:16.581 Let's go back to this thing. 21:19.501 --> 21:34.347 This was pretty much this. Now let's go to this other second example. It's very very similar, basically the same. but it uses a different 21:34.407 --> 21:47.936 image and this is actually a real world image from the so-called cellular open data sets so this is one of the privileges of having a PhD at 21:47.976 --> 21:56.121 RITMO that you get to work with very different people so some of the people in my group are actually 21:59.843 --> 22:08.927 spending a lot of their time trying to understand something very, very important about life and about cancer and about aging, which is called 22:08.947 --> 22:24.535 autophagy. And they can capture autophagy in images. because they can attach red and green fluorescent proteins to different uh let's say 22:24.575 --> 22:34.242 modules in the cells so when they go through autophagy the the balance of red and green are changing and therefore they can actually learn about 22:34.542 --> 22:45.348 let's say how different gene combinations will affect this process and this is like a really fundamental process so I decided to ship some 22:45.569 --> 22:58.174 example images from this open dataset, which by the way you can find on this Zenodo page and download the entire thing if you want. But if you 22:58.214 --> 23:08.947 install the package, you probably already have a few of them in there. So this images slash cellular dataset merged 8-bit time point blah blah 23:08.967 --> 23:22.835 blah blah is basically this image. okay let's now what i what i want to do until the break is to basically go through all the all the basic settings 23:22.895 --> 23:34.762 here uh and sort of this architecture of how you have uh yeah all these set up And then hopefully after that we have time to get into the more 23:34.862 --> 23:46.634 advanced examples such as HDR images, numpy arrays, headless mode, multiple apps, multi-channel synths, etc. But let's start slow and 23:46.674 --> 24:03.977 let's open this audio settings pane. I hope you can see it. yeah so let's focus on this top row there is this audio switch here so if i actually just 24:04.598 --> 24:05.159 turn this on 24:13.102 --> 24:27.007 So this is like a really 0.1 or 1.0 sonification. Let's see what we are doing here. We are taking the mean red value in this little rectangle, 24:27.127 --> 24:38.915 which is our image probe, And then we map that mean red channel value, which is kind of important about the autophagy biology process, and 24:38.935 --> 24:47.122 then map it to the frequency of this theremin with the exponent of two, which we will talk about later. 24:49.943 --> 24:59.628 In a way, the brighter the retina is, the higher the frequency is going to be. Extremely simple. Now we can check that there is actually sound 24:59.728 --> 25:10.093 output here. And then there is also this sort of little master slider with decibels. 25:11.634 --> 25:12.615 That is it. 25:15.613 --> 25:26.178 much of this first part of the notebook is about kind of walking through walking you through those settings but also showing you the uh cody or 25:26.278 --> 25:36.362 programmatic ways of switching these toggles because you remember i wanted to create something that is kind of between a web app and a library 25:36.502 --> 25:48.326 so you do get some conveniences of a graphical user interface but you can completely remote control the entire thing so you know if this audio 25:48.366 --> 26:02.372 thing is off then i can say app which is my variable that holds this app i can say app audio true and then we see that yeah it turned it on Or I can do 26:02.432 --> 26:13.238 something like app.masterVolume equals minus 24. That is in decibels. And then it's at minus 24 here. It's read and write. So you can also just 26:13.359 --> 26:26.407 evaluate app.masterVolume to get the current value of whatever it is. and yeah the next line is about making quick recordings so you can get a 26:26.627 --> 26:34.853 recording path in here or you can also programmatically set a recording path and then it will show up here and then you can toggle the record 26:34.893 --> 26:43.058 button so for example i can say let's just leave it on this recording and yeah 26:43.178 --> 26:45.540 kind of quiet um i'll make another one 26:48.939 --> 27:02.392 let's say i did this squeal sound and then i can just import the audio and the display modules of ipython and say display the audio of this 27:02.473 --> 27:04.775 recording and then 27:08.335 --> 27:21.778 This is what I just did. This is meant for quick prototyping. When you set up something, you can make very long recordings with it as well, 27:21.958 --> 27:31.721 hopefully. This is a quick and easy way to record your real-time interaction with There will be also another way to do this 27:31.801 --> 27:44.568 non-real-time, but we will get to that. And the other large thing here, again, sorry for the graphics experience, is this master envelope 27:44.728 --> 27:55.974 card. And this is your traditional ADSR envelope, so attack, decay, sustain, and release. There is a little drawing here that shows you 27:56.014 --> 28:09.120 these breakpoints. So now it's like 0.1 second here, 0.1 second attack, 0.01 second decay, sustained level of 1, and then release of 0.1 second. 28:09.180 --> 28:22.127 And you get this auto-computed duration that your entire envelope is going to be 0.21 seconds. So this is like when I press... the uh the mouse 28:22.227 --> 28:35.502 and activate the probe it turns red that is when we execute the attack and decay parts and stay in the sustain and then when i release the mouse then 28:35.703 --> 28:49.914 we get through the release part so Just some examples of, of course, how this can affect the sound. So I can say here app.masterEnvelope.attack 28:50.194 --> 28:58.460 and then value or .release and then another value. So now we made it a little bit longer. So I can do like, 29:02.904 --> 29:17.316 et cetera. We can make it more percussive, pointy. You see a little visualization here. 29:18.777 --> 29:33.622 And we can also put it back to where it was with this neutral, really just a fade in fade out thing. okay great yeah that's pretty much it you can also 29:34.422 --> 29:45.466 currently just read only the sample rate and buffer size but this will change later so you see we are at 48 kilohertz and 480 buffer size at least 29:45.566 --> 29:56.320 on this machine But that's all about these audio settings and again I'll try to keep these basic stuff fairly fast so we have more time for the 29:56.360 --> 30:02.289 advanced stuff. So let's go to the display settings which is the other pane here. 30:05.828 --> 30:21.693 To avoid a lot of scrolling, I can, by the way, just say app.ui and kudos to API widgets, it will create a synced copy of my entire UI. So as you see, if 30:21.793 --> 30:30.818 I open audio settings here, it's gonna be open here too. And if I close it here, It's going to be closed here too. This is like a really nice and 30:30.978 --> 30:40.647 potentially powerful application in API widgets. But yeah, it also saves us some up and down scrolling for this talk. 30:43.830 --> 30:53.894 Sometimes, when we work with images, they come with some kind of skew or some kind of tint that we want to remove. This is especially true in 30:54.034 --> 31:05.658 biology, where maybe some initial conditions on the experiment result in some strong taint. In this case, there is a quite strong green tint on 31:05.678 --> 31:18.681 the image, for biological reasons. But maybe when we look at the images, we want to remove this bias. So there are two checkboxes here. One is 31:18.821 --> 31:28.343 normalized display and another is global normalization. They have their corresponding props as well. So if I click here, you see that it 31:28.403 --> 31:34.004 scales each color channel independently to 0 to 55, which is the 8-bit max. 31:37.904 --> 31:47.186 and therefore removes the green bias. But if I say global normalization, then you guessed it, the green bias is back because now 31:47.306 --> 31:56.428 we are actually seeing what is the total highest value in all the Coral Channels, and that's what we will use as the maximum for our scaling. So 31:56.488 --> 32:05.050 in that way, of course, it will bring back the green haze. So most of the time, you don't need this checkbox. Most of the time, you just want this. 32:05.850 --> 32:16.215 And when we go to the HDR images, you will see how actually useful that this checkbook can be. We shouldn't worry about the channel offset and 32:16.255 --> 32:23.939 layer offset at the moment, because we only have a single RGB image loaded, so there is nothing to offset. But hopefully they will become 32:24.059 --> 32:38.435 interesting later. uh okay let's move to probe settings um so this is what defines the interaction at least at the moment in pixasonic so you 32:38.855 --> 32:51.608 have your image you have your whole sonification pipeline loaded And then you interact with the image using the probe. And it is technically 32:52.029 --> 33:02.278 the sub-matrix of the probe that gets passed into feature extractors and that will be sonified later on. The probe is really your way to use a 33:03.219 --> 33:18.931 kind of spatial filter on your image, sort of intuitively. It has a width and a height, so you can set it to... These are in pixels, so by default the 33:19.071 --> 33:31.460 app is 500x500 pixels. And I kind of forgot this detail early on, but currently the resize is every image that you load to this. So you have to 33:31.520 --> 33:42.006 set as a creation argument what height and width you want, and then that's what you get with that particular app object. So now we have this 33:42.046 --> 33:54.510 slider from one pixel to 500 pixel. And then that means now if I click on this probe, it's gonna be this slice of the matrix that we send to the same 33:54.570 --> 34:09.695 feature extractor. And this way you can make like, you know, scan lines horizontally or horizontally. vertically like so but you can also 34:10.775 --> 34:19.019 let's say there is an example here somewhere where I can just programmatically I already maybe know this image a little bit so I can 34:19.099 --> 34:30.205 say probe width is this probe height is this probe X probe Y set also in pixels all of these are in pixels and And now you see, I hope you see, that 34:30.846 --> 34:44.038 we have the probe precisely around this nice bright cell in the middle here. There will be ways for you to remote control the probe, not just in 34:44.078 --> 34:56.129 real time, but for non-real time scenarios as well. But for now let's move on and go through interaction modes. what I have shown now so far is 34:56.169 --> 35:11.343 the so-called hold mode which refers to the fact that you have to hold down keep holding down your mouse button to keep the probe active so when 35:11.383 --> 35:23.352 the probe is inactive or muted then it has a yellow color and when it's active or unmuted then it has a red color and so in hold you have to 35:23.412 --> 35:36.178 actually hold down the mouse to activate it but maybe you don't always want this so there is another mode called the toggle mode where now I can, 35:36.279 --> 35:47.492 if I set it to toggle, I won't have a sound. I won't activate it by just holding it down. I actually have to double click to enable it. And now I 35:47.532 --> 35:59.283 can move it around with single clicks. So for example, I can maybe comfortably I mean in this very simple didactic sonification example I 35:59.303 --> 36:11.050 can maybe look at different cells and maybe based on the pitch I get an idea of the level of autophagy that they are going through. Yeah, but 36:11.090 --> 36:21.035 just so you know for later, there are these two interaction modes. And then finally there is a checkbox here, probe follows idle mouse. The 36:21.175 --> 36:32.942 idle mouse is a little bit technical term, but it basically means you're hovering mouse without clicking. So if I check this, then I don't 36:33.022 --> 36:43.329 actually have to click. Now it just follows my mouse all the time. And you also get these little readouts the of the actual probe x and probe y 36:43.369 --> 36:52.478 coordinates and again all of these there are examples in the notebook for you later to check out how all of these parameters are actual 36:52.558 --> 37:05.844 properties of the app object so you can just remote control everything These are all the settings we went through. Now, before the break, I just 37:05.904 --> 37:15.407 want to very quickly go through the basics of feature stints and mappers, and then we are pretty much done with all the basics and we can 37:15.447 --> 37:32.614 get to the more hairy stuff. uh so yeah um let's scroll down a little bit i will make another ui instance here in the feature section so if we open 37:32.634 --> 37:48.082 this guy we see that we already have this mean red feature which is going to look at the probes matrix look at only the red channel, that is the 0th 37:48.162 --> 37:59.812 channel in RGB, and then reduce the probe matrix into the mean value according to the channel dimension. So it will reduce height, width, 37:59.972 --> 38:10.440 and all the rest. I will talk about that later. You also see that it has a minimum and maximum and number of features for you. And then as I move 38:10.480 --> 38:22.849 this, it still follows my idle mouse here. So as I move this probe around, it constantly gives me the current red channel pixel values or the mean 38:22.929 --> 38:34.767 pixel value within the probe. So you see the minimum and maximum in the entire picture is 25 and 255 and then this is my current value You have a 38:34.988 --> 38:40.270 reset button and the detach button. So just quick recap in the beginning 38:42.991 --> 38:54.546 Yeah here this is where we created this so we created this mean channel value here we filter for the red channels only this will by the way accept 38:54.626 --> 39:08.279 any slicing syntax but more on that later And then we attach it to the app. And so in this moment, in the moment of attachment, the feature actually 39:08.359 --> 39:19.541 gets to see the entire matrix, the entire picture in the app, so that you can compute useful stuff like global minimum and maximum. But we can 39:19.621 --> 39:31.223 also, if we click on the reset, now we see that it's reset to the current value. and now we have a running minimum and maximum so this might be 39:31.263 --> 39:45.508 sometimes useful when you maybe want to put it on a cell like this then you reset and then you kind of click around a few cells to get some kind of 39:45.668 --> 39:57.813 range that kind of only describes these cells for example And what is the dangerous red button? It's called detach. So as we attached to the app, 39:57.933 --> 40:13.257 we can also detach from the app. So if I click on this, oh no, our feature is gone. It's dead. uh no you know it isn't that um if i scroll down here uh i 40:15.058 --> 40:26.444 can see that it uh it is still alive if i bring up its ui so you remember the user interface of the app was app.ui And if you say min-red, where our 40:26.704 --> 40:37.252 feature object is, min-red.ui, and it's here. And it's still alive, in fact. Even if I reset it, it will still work. This is, again, kudos to 40:37.332 --> 40:48.401 iPyvegets that I can have synced copies of these different UI elements scattered around the notebook. It can be quite convenient. But now, if I 40:48.441 --> 40:55.264 want to attach it back, I can just say app.attach and watch what happens to the minimum and maximum. 40:58.023 --> 41:06.288 again at the moment of attachment it gets to look at the whole image once and we will see how this can be useful when we will create our custom 41:06.788 --> 41:17.255 features by inheriting from this the feature base class okay let's create another feature which let's say looks at the main channel value 41:17.395 --> 41:28.881 but across RGB so it will return three values so it's gonna be pretty much the same but without saying this part So it's going to be meanPix, a mean 41:28.921 --> 41:41.904 channel value. This is basically a convenience abstraction with the name meanPix. And then I say, and I create it. And then, yeah, a little bit 41:41.944 --> 41:55.486 about how the filter is working. So actually when you're reading an image and that's what this kind of 4D was about in this diagram, So 41:55.686 --> 42:05.110 actually internally all images are stored as high-thread channels and layers, which means you can have arbitrary number of color channels and 42:05.170 --> 42:13.773 an arbitrary number of layers. This can be perhaps useful for using it with hyperspectral imaging where there are let's say 128 color 42:13.793 --> 42:29.383 channels. or MRI slices which you can either stack in the channels or the layers dimension let's say you can have a time lapse of MRI DICOM volumes 42:30.184 --> 42:40.834 where you put the spatial slices on the channels dimension and then you put the temporal slices on the layers dimension So every image is stored 42:41.334 --> 42:50.727 in this way. So when we are saying filter channel 0, we are actually filtering the channel's dimension to have only the 0th element in 42:50.767 --> 43:03.591 there. So if I don't do that, if I don't say that, it's going to give me the entire image. Okay, enough talk. Let's attach this. And before we get a 43:03.811 --> 43:12.815 scroll back to the user interface of the app, there is the features prop of the app, which will actually give us a list. So now we see that there are 43:12.875 --> 43:26.173 two features currently attached to this app. A feature with this ID called mean red and another feature called minpix and lo and behold now 43:26.414 --> 43:41.738 we have two little cards in here and if we move this thing around we see that in fact this minpix has three features it gives you the channel wise 43:41.798 --> 43:55.596 minimum and maximum and gives you the current value as well that's i think all the basics about features Now we kind of took out information 43:55.656 --> 44:04.762 that we care about from the image. Now we have something to sonify it with. So that's going to be a synth, some little gadget that makes sound. 44:04.883 --> 44:17.091 And up there, we created this theremin here that we imported from the Pixasonic synths. And we attached it to our app. And the theremin has 44:17.171 --> 44:29.140 some creation arguments that can... defined basically default parameters but we I don't have to worry about that now you can see them if I 44:29.640 --> 44:39.588 do this reset so it has a frequency parameter amplitude depending so if I turn this guy on even though now actually the mapping is active so as you 44:39.648 --> 44:46.748 see this frequency slider with move around as I'm 44:47.709 --> 45:00.052 interacting with the image. But I can also just grab the sliders and maybe test different parameter combinations for myself. And then I can 45:00.092 --> 45:16.881 reset it. And as you know, we can also detach it. So if I detach it, it's removed. There is another just maybe interesting detail here. If I 45:16.921 --> 45:27.989 create another theremin like this, I don't attach it to the app. I just pull up its own UI and now it won't make a sound because it's not attached 45:28.009 --> 45:38.987 to the global audio graph. that is here so normally when we attach a synth to the app when then we connect to the global audio graph like that but we 45:39.027 --> 45:51.202 can also just go around this we don't have to have an app and we just attach this guy directly here and Here is now a little bit of advertisement for 45:51.242 --> 46:02.088 the awesome signal flow library that my library Pixisonics is based on. So this is a fairly new Python library for doing real time and non real 46:02.128 --> 46:13.658 time audio. and it's made for Python, so it's very Pythonic in its interface, and it's really nice written in C++ from the ground up with 46:13.738 --> 46:26.468 very few dependencies. So if you know Signal Flow, I plan to put in more and more features so that the stronger your Signal Flow skills get, the 46:26.508 --> 46:36.656 more you can also get out of my library. but basically in signal flow we create this global audio graph like this, and that's where we attach 46:36.916 --> 46:47.584 things. So you can do the same thing here without the app. The app.graph prop will hold a reference to this global audio graph, so we can just get 46:47.624 --> 47:01.230 it like this, and then we can say graph.play . And then now we can actually interact with these sliders. And then I can say graph.stop, also signal 47:01.250 --> 47:11.937 flow 101. Then it will remove it from the global audio graph. Just something that you can be aware of. So now we have this synth. We have a 47:11.997 --> 47:23.447 feature. We have to put a mapper in between, and we have to decide what feature we map to what synth parameter. Again, this is what parameter 47:23.467 --> 47:38.458 mapping sonification is all about. The good thing is that more mappers can read from the same feature, And many different mappers can 47:39.919 --> 47:52.845 influence different parameters on the same synth. So the whole thing is kind of modular in this way. So now, I think I did show all these things. So 47:52.885 --> 48:02.170 you remember when we created this mapper, this is so that I don't have to scroll up there, it was red to freq, a variable, mapper, and then we 48:02.230 --> 48:12.855 passed first, the first required positional argument is mean red, which is going to be the variable to our feature that was focused on the 48:12.915 --> 48:22.121 red channel. And then the second positional argument is the theremin frequency. This is actually going to be a dictionary so we can just use 48:22.161 --> 48:32.389 this syntax our theremin variable with the theremin synth in there and then we say at frequency which is the frequency parameter and if you look 48:32.429 --> 48:42.997 at here this is actually a dictionary with many different things we don't really have to look at most of them what is what i want to highlight 48:43.057 --> 48:51.565 is the min and max that is already in there, which is going to be interesting in a second. And then we also have this exponent and name, 48:51.805 --> 49:04.658 which we'll also get to. So how did we get this mapping done without ever saying what are the input and the output ranges? We just said, hey 49:04.738 --> 49:13.407 mapper, here is this feature and please map it to this synth. And that was somehow enough. Well, it was enough, because if we don't say anything 49:13.487 --> 49:23.938 about the input and output ranges, then every feature will by default report its minimum and maximum. So if we don't tell anything to the 49:23.978 --> 49:27.662 mapper, it will just say, okay, what is your current minimum and maximum 49:30.245 --> 49:40.568 at this feature? And this is also where potentially the running minimum and maximum that I showed earlier can be actually musically relevant, 49:40.689 --> 49:53.873 or at least creatively relevant, that you get this sort of morphing input and output range on your mapping. And then each parameter at each 49:53.973 --> 50:05.100 synth in Pixisonics reports a sensible minimum and maximum. It's not set in stone, so you can easily override this, but if you don't say 50:05.180 --> 50:15.305 anything, you get some sensible defaults, at least that was the idea. So since we didn't say anything, this is actually the spelled out version 50:15.346 --> 50:24.814 of how we created this mapper. So we said, Object in is mean read, object out is term in frequency, and then in low, in high, none, none, out low, 50:24.914 --> 50:34.905 out high, none, none. This is why it's automatic now. And when we say exponent and clamp and etc., we will get there. We can also look at these 50:34.985 --> 50:45.251 things. And then we see, even though we set them to none, they now actually show the truth, sort of what is the current input and output 50:45.311 --> 50:57.878 ranges. And these are now showing up as numpy arrays of shape 1, 1, but they are just a single number right now. This is going to be important 50:58.099 --> 51:08.407 when we are mapping from features that have, let's say, 50 different features or they report a feature vector of 50 or something like that. 51:10.766 --> 51:23.531 just to illustrate i can explicitly set these ranges like this so if i get back to my sonification here now we will see that most of the time i'll 51:23.571 --> 51:38.107 just go to oh yeah i have to actually attach my theremin back yeah thank you Yeah, so now almost always it will sort of clamp to the minimum and 51:38.147 --> 51:46.971 then when I go to a very bright place it will suddenly chirp up to the maximum because we actually defined a very narrow range of pixels for 51:47.011 --> 51:59.344 this input range. But of course, if I put it back to none, it will just keep looking at the features minimum and maximum, and then we get this sort of 51:59.504 --> 52:10.971 default behavior that we had earlier. Yeah, clamp is to basically, in this previous example, you've seen that it didn't want to explode in a 52:10.991 --> 52:22.437 way. So even though I set up a narrow frequency range, it didn't extrapolate the mapping and go very deep under or go in a very high 52:22.477 --> 52:32.044 frequency according to this mapping, following the mapping curve. you can turn this off by saying clamp false but be a bit careful with it 52:32.084 --> 52:46.616 because things can explode and then for the exponent i made this little interactive plot where you can the underlying thing the underlying 52:46.696 --> 52:57.582 mapper function will be the same one that is used in the mapper class so if you move this slider you get the sense of what is the mapping curve of your 52:57.762 --> 53:09.486 input output range and for those of you familiar with MaximusP it's exactly what the scale object does so if you're familiar with that then 53:09.946 --> 53:20.487 you get a head start so this is the mapping curve that we are currently using with our mapper and you can also of course get that if you evaluate 53:20.527 --> 53:34.253 the exponent you see that is true now we will go to a little bit more advanced examples involving a bit more code and i don't know if we will 53:34.273 --> 53:45.959 get to see all of these i really hope so so maybe i will be a bit more sparse with the code explanation You can, of course, read these things a bit 53:46.020 --> 54:00.128 more calmly at home with a cup of tea. But yeah, now I will just briefly go through what the code does. And as you see, there is no more blah blah 54:00.188 --> 54:09.394 around the code cells as well. That's another thing that I have to catch up on. So sorry about that. I will be your live documentation this 54:09.514 --> 54:21.916 evening. And then later on, it's gonna get better, but it's very fresh. So what we will go through now are what I wanted to implement as sort of 54:22.036 --> 54:33.559 breakout doors. So you can hopefully easily integrate Pixisonics in your other workflows and maybe customize it to your needs. So first of 54:33.599 --> 54:43.242 all, until now, only looked at some extremely extremely simple sonification so like mapping the red color channel value to the 54:43.302 --> 54:53.106 frequency of a sine wave it's like it cannot get any simpler than that i guess um but now we will look at some more uh some use cases that more 54:53.246 --> 55:06.272 connect to i would say real world use cases so the first big enabler is loading numpy arrays and also loading hdr images Then we will briefly 55:06.332 --> 55:17.497 look at non-real-time rendering. Then we will look at headless mode, which lets you kind of completely go around the UI and also use 55:17.597 --> 55:28.622 Pixisonics in a script. Then we go to multi-channel synths, multi-channel expansion in signal flow basically, and how you use it in 55:28.682 --> 55:40.589 Pixisonics. Then I think the sort of coolest part of this whole thing will be the example that I have for the very end, which is setting up a 55:40.749 --> 55:50.655 custom feature class that inherits from the feature base class. And this example, I will set up a K-means classifier on the image domain, and 55:50.695 --> 56:07.561 then map the latent or, you know, the histogram of class presence within the probe, according to the clustering, to the harmonics of harmonic 56:07.601 --> 56:19.343 tone. That will maybe give us some intuition about how different cells are perhaps related to each other. But we will get there. For the 56:19.383 --> 56:28.566 advanced examples, there are a few imports that we have to do. All of these are actual dependencies of Pixisonics so you don't have to 56:28.666 --> 56:41.252 install anything extra. It's gonna be pillow for loading images. It's gonna be NumPy because of course, and yeah, IPython display. So I'll do 56:41.272 --> 56:54.938 that. Okay, so now we will see some real good use of the normalized checkbox. So if I create an app, as you see it just creates this blank 56:55.038 --> 57:08.307 thing here and now i can there are also some 16-bit images in this cellular data set i mean i should say there are thousands of them but a few 57:08.347 --> 57:18.994 of them are included in pixisonics as just sort of test test images so if we give it this path this is of course relative to the location of the 57:19.035 --> 57:32.428 library then we can load this image, which is a 16-bit image of the same thing. But I can barely see anything. So here is where the normalize 57:32.488 --> 57:44.019 button is king, because now we can actually see the image. This is because in the biology experiments, the microscope might not actually 57:44.139 --> 57:55.508 fill the entire 16-bit number range so the images might come off unnormalized and then you can easily normalize it for yourself an 57:55.568 --> 58:04.175 important thing that I forgot to say in the sort of beginning part is that all these display settings are only affecting the display they won't 58:04.295 --> 58:15.701 like rescale the numbers that you are sonifying or something it's only for you for the display So that was loading an image file that is a 16-bit 58:15.741 --> 58:25.563 image and this is kind of kudos to the pillow library that I can handle it like that. But maybe you have some very specific custom data and the 58:25.603 --> 58:37.709 common experience in Python that in the end everything will probably come down to NumPy. or at least be interoperable with it. Here is an 58:37.809 --> 58:50.337 example of, of course, using the pillow image class. This is pretty much what I do internally in the app class anyway, but now you can see how it's 58:50.397 --> 59:05.369 done. This is the earlier 8-bit RGB image. and you see that I open it here, create a numpy array out of it here, and instead of saying app.load image 59:05.449 --> 59:18.960 file with a path, I'm saying app.load image data with a numpy array. and again this numpy array can be up to four dimensions so obviously it will 59:18.980 --> 59:26.765 be at least two dimensions height and width and then it can have arbitrary number of channels and arbitrary number of layers so maybe 59:26.785 --> 59:36.713 let's get into that so here is an example where i have two 16-bit images one for the red channel the red fluorescent protein and the other one is 59:36.733 --> 59:48.658 the green fluorescent protein And I just separately open all of them with Pillow, stack them on the, well, the last axis will be the channels, 59:48.738 --> 59:59.663 the third axis here, or the second, because these are just heightened with two-dimensional arrays up to this point. and then we load it like 59:59.703 --> 01:00:14.008 that so you see what we load is height width and then two channels so if we come back here and now we see that we get proper colors and yeah again if we 01:00:14.048 --> 01:00:28.555 turn off normalize it might not show anything but yeah if you don't see anything try to normalize it um yeah um let's take this to the next level 01:00:28.735 --> 01:00:38.241 so oh yeah uh one thing that i maybe wanted to yeah uh let's let's make use of these channel offset and layer offset sliders that i showed you 01:00:38.281 --> 01:00:49.286 earlier so now they're disabled because again the shape of this volume is height width and channels and there are only two channels so if the 01:00:49.306 --> 01:00:58.951 channel offset kind of implies that you have at least four channels otherwise it's disabled so because up to three channels we can 01:00:58.991 --> 01:01:12.178 visualize in one go okay now i will um so this this data set is actually time lapses of images uh going through autophagy and also images uh 01:01:12.358 --> 01:01:23.104 drosophila s2 cells so fruit fly blood cells going through starvation induced autophagy and there are five steps in the time domain here so we 01:01:23.144 --> 01:01:32.610 can actually uh we will end up loading all of these into one volume And then the cool thing about that, hopefully, is that then our image probe 01:01:32.951 --> 01:01:44.939 will be able to see through all the dimensions at once if we want it so. So here an example is how we stack the time lapse spots on the channel 01:01:44.979 --> 01:01:54.646 dimension, which semantically does not make too much sense, but just to showcase the channel offset sliders. So now we see the first three time 01:01:54.706 --> 01:02:07.035 steps as an RGB image. And then we can offset it like this. Of course, this is also remote controllable. So imagine if you have a hyperspectral 01:02:07.115 --> 01:02:19.322 image of something of 128 channels, then you will be able to kind of scroll through this and see a different snippet. what makes perhaps a 01:02:19.382 --> 01:02:29.907 little bit more sense is to stack a time lapse on the layer dimension so here I'm adding this extra dimension when I'm making it into an array and 01:02:29.947 --> 01:02:44.197 then therefore the stacking will actually happen on the fourth axis so Now we are having this kind of shape. So one channel image, but five 01:02:44.437 --> 01:02:58.149 layers. And then it shows, of course, as a grayscale image. And now we have the actual time lapse displayed like this. great and then another 01:02:58.309 --> 01:03:11.103 example about this now we can combine this you know both red and green channels and all the and all the time steps like this so we will see two 01:03:11.163 --> 01:03:25.014 channels and five layers so it is going to be now in color basically and as you see it gets a little bit more yellow because the red channel is 01:03:25.054 --> 01:03:33.983 getting stronger and stronger and that is actually the little autophagy vesicles that so that basically shows that the thing is 01:03:34.944 --> 01:03:44.693 having higher and higher levels of autophagy Okay, but yeah, one more, maybe I won't show this example, it's basically the same thing again, so 01:03:45.894 --> 01:03:58.286 instead we also add one more channel, so it's going to be three channels and five, but I hope you get the gist now. From now on basically we will 01:03:58.326 --> 01:04:09.891 just copy this snippet into all the following examples and this is going to be our basis. Now we create another app, so we will have two apps in 01:04:09.951 --> 01:04:26.557 parallel now, app and app underscore two. And as you see, we will just copy this thing again. And of course, I have to normalize it so I'm able to 01:04:27.538 --> 01:04:38.285 see it properly. And then non-real-time rendering. So basically, while Pixisonics is trying to be interactive and real-time and 01:04:38.526 --> 01:04:50.568 immediate, There are also ways to basically script the movements and the size or dimensions of the probe with timestamps and then render that 01:04:50.728 --> 01:05:03.110 out as a sonification of whatever pipeline you set up in there. So let's create a minimal setup here. So this is going to be the same boring 01:05:03.190 --> 01:05:14.833 example, sorry for making you listen this. simple channel value, simple theremin and simple mapper pretty much the same thing except now 01:05:14.893 --> 01:05:24.855 it's on the 16-bit data so in fact yeah one more cool thing I could show is that now when you go into the feature it actually shows you the 16-bit 01:05:26.655 --> 01:05:40.982 numbers from the image while of course this visualization is only RGB 8 bits yeah so now what I can make is what I call a timeline so it's going to be 01:05:41.002 --> 01:05:54.893 a list of tuples where each tuple is a number which is interpreted as seconds the time point in seconds and then the props basically for the 01:05:54.933 --> 01:06:05.762 probe so you remember there were these props app dot probe width app dot probe height etc etc And now we can pass these. So see here, I set up a 01:06:05.982 --> 01:06:17.213 horizontal scanning, this vertical line width of one pixel height of the entire image. And then I set the probe XY to 00. And then at the 01:06:17.253 --> 01:06:29.200 duration, which is five seconds, the probe X should be at the last pixel. So this will create this linear interpolation between these two 01:06:29.260 --> 01:06:39.286 numbers for these five seconds. And then this is my timeline. I give it a target file name. And then I'm saying app, render timeline to file, my 01:06:39.306 --> 01:06:47.551 timeline, and then target file name. And then here with the IPython display, I display the results. And so it looks like this. 01:06:53.312 --> 01:07:06.196 Not very interesting sonically, but hopefully you get the gist. Here is another example where we want to do like one minute. And then you can just 01:07:06.316 --> 01:07:16.520 do it exactly like the previous one, but like editing the duration to 60. But I just show you this as a potential optimization step that you can put 01:07:16.660 --> 01:07:28.825 in. So the app actually has a frames per second that is normally controlling the interactivity of the real time graphics so that it 01:07:28.905 --> 01:07:39.251 never goes higher than a certain frames per second. But when it comes to non-real-time rendering, the FPS is going to be actually the number of 01:07:39.491 --> 01:07:49.498 times the entire pipeline should be evaluated. So all the features, all the mappers, and all the synths. So in this case, I take it down to 10. So 10 01:07:50.299 --> 01:08:00.145 times a second, the entire sonification pipeline will be evaluated and then rendered into a 60-second file. And then after I'm finished, I'm 01:08:00.225 --> 01:08:10.452 putting it back to the default 60 hertz. So here is my one minute very slow scanning through that image. 01:08:13.174 --> 01:08:25.460 Hopefully this gives you an idea how to make maybe much longer sonifications like that. Next somewhat adjacent topic is headless 01:08:25.500 --> 01:08:36.391 mode. So far I always showed everything with UI and of course again Pixisonics is made with interactivity and audiovisual 01:08:36.431 --> 01:08:44.418 representation in mind and especially given that it's in a Jupyter notebook. But let's say if you want to use it in a script, let's say you 01:08:44.458 --> 01:08:54.303 have 5,000 images and you maybe have this same scan path that you devised for them, and you just want to create some kind of weird sonic 01:08:54.363 --> 01:09:07.109 fingerprint in a for loop or something like that, you can do that in headless mode. So an interactive example here is that in this app three, 01:09:07.329 --> 01:09:20.012 I say headless equals true when I create it. We don't see the UI anymore because it doesn't have one. Actually, it still has one, but it doesn't 01:09:20.052 --> 01:09:30.915 show it. Now we loaded an image. Again, the same kind of minimal sonification example. And here I actually can print that, yes, I 01:09:31.075 --> 01:09:44.999 managed to attach those things to my app. I can get where my probe is and how big it is at the moment. I can move it to a certain place. I can enable 01:09:45.019 --> 01:09:58.543 audio processing on this app, in this subgraph. I can even print the global audio graphs status. This is a bug that's hopefully going to be 01:09:58.584 --> 01:10:13.331 fixed. Don't worry about that. This device is only 16 gigs of RAM, so you know. and now you remember when I clicked on the probe then I essentially 01:10:13.831 --> 01:10:28.340 unmuted it so I turned this unmuted prop to true so now if I do this now we should hear like 60 Hertz sounds but if I move the probe now to a different 01:10:28.420 --> 01:10:40.166 place now it changed into a different sound and then if i call this unmuted false you will hear that the release part of the adsr will also be 01:10:40.206 --> 01:10:56.298 called so it just fades out so uh a little bit silly example is to um in a while loop move the probe uh while it's on so here we unmute it then we kind 01:10:56.318 --> 01:10:59.401 of move it to a place and then we mute it again 01:11:02.990 --> 01:11:13.259 And this is just that test image from the very, very, very, very beginning, this UV map here. So that's why it's just a linear ramp in 01:11:13.359 --> 01:11:28.252 frequency, or actually exponential. But where were we? Yeah, if I do it in another image, it will, so I load a different image, but keep 01:11:28.333 --> 01:11:38.833 everything the same. Now, of course, it will have a different melody. Great. So here is an example how we would do this in a loop, let's say in a 01:11:38.893 --> 01:11:50.376 script. So let's say I have this folder of images, load all of them, create this sort of prototypical timeline for myself. Yeah, this is 01:11:50.676 --> 01:11:58.658 just a little initialization so that I actually have a global audiograph. And the interesting thing is now I can just create these 01:11:58.878 --> 01:12:11.120 things once, and then I will one by one attach them to the various apps that I create. So here I create a context. So I create this app in headless 01:12:11.280 --> 01:12:23.443 and in non-real time as the app variable for this little context here. And then I load an image file, attach these guys, the feature, synth, and 01:12:23.463 --> 01:12:36.388 the mapper. render the timeline and then display it so this is going to be five different images and now we have 01:12:39.366 --> 01:12:50.269 different scan paths or like the same scan path rendered on different images. And the same thing, this is just to say that you don't have to 01:12:50.329 --> 01:12:59.572 render to a file, you can also render to a NumPy array. So if you say instead render timeline to array and just pass the timeline, then you 01:12:59.592 --> 01:13:05.097 get the NumPy array back. which is basically the waveform like the 01:13:07.079 --> 01:13:18.551 time domain representation so if I evaluate this the audio can still take an umpire array it's just then you have to tell the sample rate and 01:13:18.591 --> 01:13:33.161 then it will work the same Maybe this can be useful for some cases. The last two cool examples will be one is the multi-channel expansion for 01:13:33.281 --> 01:13:46.128 Synths and the other one will be the custom features specifically this k-means example. And then we are done. So let's create app number four 01:13:47.248 --> 01:14:01.716 here. Load the same image data. Normalize it just so we see. So here I get a little bit experimental and I'm saying instead of reducing on the 01:14:01.756 --> 01:14:17.127 channel dimension, I will now reduce on the row dimension. So the feature will look at the four dimensional volume and it will take the 01:14:17.187 --> 01:14:35.513 mean across all the axes and keep just the column. So the target dim is the height. So we add this And then we will see that our feature now has 50 01:14:37.615 --> 01:14:54.455 features. It was 500 because it initialized with the whole image. So now we see what is basically the mean in the vertical dimension. and without 01:14:55.716 --> 01:15:06.948 adding the column dimension as well I will now create an oscillator which is kind of a more beefed up version of the theremin with different 01:15:07.228 --> 01:15:22.319 waveforms low pass and high pass filter and some other things And then I create a list of frequencies, and this is going to be just a copy of 440 16 01:15:23.400 --> 01:15:35.967 times. But since I give it this way, it will actually create 16 voices or 16 channels for the synth. And when I say panning , it will linearly 01:15:36.028 --> 01:15:48.096 interpolate it so that it also becomes a length of 16 lists. So this is my way of simply spreading them in the stereo field. Waveform is going to be 01:15:48.156 --> 01:16:02.547 a saw. And then, yeah, let's go. and then finally there is this mapper here so I'm saying this mean row should go to the OSC banks frequency yeah 01:16:03.368 --> 01:16:06.230 let's do that and then now 01:16:10.533 --> 01:16:14.456 if I enable audio here yeah 01:16:18.556 --> 01:16:26.019 So one idea here is that you can create this little scan path for yourself and then kind of... 01:16:37.582 --> 01:16:48.348 it could be cool so now that we can kind of see through all the time steps I mean we cannot see them visually we can only see one at a time but our probe 01:16:48.428 --> 01:17:00.035 can actually see through all of it so it could be useful for example to find cells that we don't see on the first picture so if i go to this empty 01:17:00.095 --> 01:17:13.840 space it's mostly just the slightly detuned minimum frequency value in the range of course if i go over something bright it will get bright but 01:17:13.920 --> 01:17:18.381 what if i let's say just reduce it a little bit 01:17:21.503 --> 01:17:24.484 and then just kind of go over these areas 01:17:31.252 --> 01:17:43.600 okay there is something here so this is totally not rehearsed if i now come here i can see that i actually found this little cell here only based 01:17:43.680 --> 01:17:57.349 on the sound the amplitude is stationary so here we see all of them are 0.5 but as i move this thing we see that their frequencies or well hopefully 01:17:57.369 --> 01:18:11.659 you see that their frequencies are changing according to, well, the feature value, which is going to be the mean brightness value, but 01:18:12.340 --> 01:18:23.347 reduced to the column dimension. And the cool thing about this is that the mappers can dynamically resample between different lengths. So 01:18:23.807 --> 01:18:38.298 even though I have a fixed 16 voice OSC bank, I can make the probe a little bit larger, let's say 65, and now it properly adjusted to 65 features, 01:18:39.019 --> 01:18:49.147 but then the 65 values will be kind of resampled linearly into 16, and then the mapping will be done to the 16 sawtooth waves. 01:18:52.649 --> 01:19:01.917 and stereo panning is basically you go from the top to bottom so imagine here at the top is all the way left and here at the bottom is all the way 01:19:01.997 --> 01:19:15.507 right according to how we actually set it up but again you could use another mapper to attach to the panning or amplitude or all these cutoff 01:19:15.608 --> 01:19:28.015 frequencies of the low pass or high pass filter or the resonance etc etc so I guess we don't have time to move into all that but yeah maybe if I just 01:19:28.115 --> 01:19:45.059 go a little bit crazy and add the other dimension as well and then make this a little bit wider something like that Now we see that we have two 01:19:45.379 --> 01:20:01.763 features in here, the mean row and the mean column, and two oscillators here. And then maybe this can be interesting as a toy or some kind of 01:20:02.204 --> 01:20:14.565 experimental sound design interface. So you can strum one little cell, like a little string or something like that. I don't know. For me, this is 01:20:14.585 --> 01:20:28.137 fun, but hopefully it's educational as well. Yeah. Let's go to the last example for today, which I think is hopefully maybe the most sort of 01:20:28.277 --> 01:20:37.805 instructive or inspirational, which is how we can make our totally custom features instead of these boring, mean, and... Oh, by the way, I 01:20:37.845 --> 01:20:50.046 kind of forgot to tell you that. all the basic kind of NumPy reducers are built into this so you can say the reduce okay let's let's go there first 01:20:50.126 --> 01:21:03.698 so let's create this fifth app here load the same data same example thing and then Let's look at this base feature class. So far, we always use this 01:21:03.778 --> 01:21:12.422 mean channel value or something like that. But actually, the base class has it all. All of these other features are just convenient 01:21:12.442 --> 01:21:23.486 subtractions with different configs for this. So what do we have with the basic base feature class, which is called feature, is filters that 01:21:23.526 --> 01:21:37.383 will let you use any slicing syntax. We have a target dimension that we reduce to and a reducing method. So we can rewrite the mean channel value 01:21:37.423 --> 01:21:48.367 abstraction like this, which is no filters, target dimension is the channel dimension and the reducing method is the mean. And this can be 01:21:48.427 --> 01:22:04.911 mean, median, min, max, sum, standard deviation, variance, I think that's about it. Yeah, let's attach this. And then here are some, I 01:22:04.951 --> 01:22:17.116 guess, examples. So if you don't say anything by default, it's going to be a mean pixel. Here is a median of the pixels. We looked at this mean row 01:22:17.156 --> 01:22:28.284 and mean column before. Here is like a median on the red channel. And by the way, these, I don't have spelled out examples of these, but what is 01:22:28.544 --> 01:22:40.234 perhaps powerful about these is that any slicing syntax goes, so you can say, You can pass a list, if you have like 200 channels, you can say 0, 10 01:22:41.254 --> 01:22:56.126 and 22 in the list and then it will filter to only those. Or you can say in a string 10 colon 50 and then it will slice on the 10th to the 50th. Any 01:22:56.166 --> 01:23:07.831 normal slicing syntax will go. If you say none, there will be no filter. and then you have four filters for the four dimensions yeah so that is 01:23:07.951 --> 01:23:20.137 about it that is the base feature class and now how how do we inherit from it so that we can create our old thing so this is the last part of today 01:23:20.157 --> 01:23:29.222 we're going to create our sixth little app here load it with the same same thing 01:23:34.974 --> 01:23:44.660 Let's look at a very didactic example. My random feature, which is going to just generate random numbers without actually looking at the image. 01:23:45.501 --> 01:23:56.928 The two methods that you want to override are the process image and the compute. You remember I told you when a feature gets attached to an app, 01:23:57.305 --> 01:24:08.647 it gets to look at the image the whole image once and this is good for doing kind of like a setup so that you can by default it will look at global 01:24:08.687 --> 01:24:21.229 minimum and maximum and that's it but you're free to do anything you want so now i just return a random matrix so the same as the incoming matrix 01:24:21.289 --> 01:24:32.292 which is in this case the whole image whole four-dimensional image And then the compute is going to be what is called every time the probe really 01:24:32.392 --> 01:24:42.460 changes. So every time you are kind of scrubbing around with the probe, it gets sent to the features, then the mappers are evaluated, which will 01:24:42.500 --> 01:24:53.969 look at the attached features, etc. So this is going to be now the probe matrix. And so here I'm just doing some other random things, and then 01:24:55.008 --> 01:25:12.607 attaching this to my app so just to see how this looks like yeah here we have a feature and then it will just generate random numbers every time i 01:25:12.627 --> 01:25:26.991 every time i click okay another variation of this that you can still use those props the filter and the target dim as in the base class so now i'm 01:25:27.031 --> 01:25:41.155 setting the target dim to zero so that's the height dimension and of course first it created a 500 length vector because it's a 500 high 01:25:41.235 --> 01:25:53.603 picture but of course now it will adapt to my probe and then it will of course generate random numbers of 50 or random vectors of 50 numbers 01:25:54.464 --> 01:26:10.178 whenever I move the probe. but now comes the perhaps more interesting use case of this so what if we set up a custom feature that will cluster the 01:26:10.318 --> 01:26:22.720 image into arbitrary clusters and then whenever we prove the image it will return the distribution of different classes so each pixel is 01:26:22.780 --> 01:26:31.700 labeled in a way but in a more nuanced histogram at least that's what this example does Here is how you could also plug in your little neural 01:26:31.740 --> 01:26:44.149 network or whatever you want. So this uses the scikit-learn package. I believe I already installed it here, but yeah, you can install it there. 01:26:45.029 --> 01:27:00.437 And so we import the canines. And then what we do here is to, in the process image, get like fit a cadence model on the features which is going to be 01:27:00.797 --> 01:27:17.532 all the vectors of colors so all the 3d or you know rgb pixels in the entire volume all the time steps all the color channels that's actually true 01:27:17.552 --> 01:27:31.015 here We predict the labels and also save the transformed image. Maybe we want something with it later. And then when we compute, then we again 01:27:31.095 --> 01:27:42.018 predict the labels of all those pixels based on the pre-fitted K-means model. And then we make this little histogram of the labels 01:27:42.118 --> 01:27:57.031 distribution within the probe and normalize it so that it sums to one. And so if I attach this, it will take a little bit of time. And then I will 01:27:57.051 --> 01:28:11.809 detach this random feature thing. so now i see that since i created this thing with 10 clusters which is you know maybe useful maybe not we now get 01:28:11.849 --> 01:28:26.915 this normalized membership values or so to say so for example 36.6 percent of pixels are members of the 10th class in this probe, for 01:28:26.975 --> 01:28:40.906 example. This could be cool, but it will be even cooler to actually hear what this means. So here I'm setting up a little theremin so sine wave 01:28:41.006 --> 01:28:51.373 essentially with 10 channels and I create a harmonic series out of this so there's this fundamental frequency of 110 and I say the number of 01:28:51.433 --> 01:29:04.299 harmonics is the number of clusters in this cadence feature and i'm setting fundamental frequency times that you know harmonic series and 01:29:04.399 --> 01:29:16.027 then i create a theorem out of this and what i will do is that i will map to the amplitude of the different partials in this and attach that to and as 01:29:16.047 --> 01:29:26.273 you see these are the frequencies so it's a proper harmonic series This is just for you to actually show what the K-means image looks like. So I'm 01:29:26.733 --> 01:29:41.980 installing Seaborn and Matplotlib to just visualize the class distributions in this thing. Maybe useful, maybe not. But I think what 01:29:42.000 --> 01:29:47.663 I'm kind of excited about is how it sounds. So if I now enable audio in here. 01:29:50.616 --> 01:29:54.659 I 01:29:55.220 --> 01:30:06.649 get a different kind of harmonic tone for each of the cells, but it's kind of hard to explain what that exactly means, because it's these latent 10 01:30:06.749 --> 01:30:16.083 clusters in the image. But it might be good to find, for example, similarities between two cells. Because if it kind of sounds the same, 01:30:16.283 --> 01:30:25.168 for example like this, I already know how this sounds because I listened to this a thousand times. So if you find like, you know, if you're going 01:30:25.188 --> 01:30:29.910 around and... 01:30:29.950 --> 01:30:33.332 Yeah, 01:30:33.352 --> 01:30:44.217 okay, we can see that it kind of is the same here. Maybe that will give me an idea to visually inspect that image there. So now if I scroll through, I 01:30:45.218 --> 01:30:53.746 can kind of see that actually this is where they are most bright. So maybe there is something about this brightness that is captured in this 01:30:53.766 --> 01:31:03.029 sound. Maybe something else. And then I can also show you this example that I showed with multi-channel synths, which is like, you know, I can 01:31:03.089 --> 01:31:14.877 just go through the image with this probe, I can see that the empty space, which was, I think, kind of this dark yellow-orangey thing here, is 01:31:14.957 --> 01:31:26.783 clustered into the same class, of course. And then it's this third, I think, partial. So anything that is different means that something 01:31:26.903 --> 01:31:40.093 else is in there that I should maybe look out for. So we hear that that very faint cell there created a different sound. And then another thing, if 01:31:40.173 --> 01:31:52.075 you remember, there was this kind of small hidden cell here, and we can very easily hear that something is there. So if I just use kind of 01:31:52.135 --> 01:32:07.299 blindly... Okay, it's kind of probably there. Oops, now that's another one, sorry. I wanted to... Let's see what's the difference here. Ah 01:32:07.359 --> 01:32:18.084 yeah, a bit more bass. The nature of Caymus is that it fits differently every time, so I kind of hear these different correspondences every 01:32:18.144 --> 01:32:32.050 time I try this example. Of course you can set your seed. But yeah, if I would listen for this kind of sound, which is maybe similar to how this 01:32:32.090 --> 01:32:32.390 sounds, 01:32:41.288 --> 01:32:41.589 Then 01:32:47.237 --> 01:32:56.826 you potentially find things that you cannot exactly see. This was for example one thing when I was talking to biologists. You know, it's hard 01:32:56.866 --> 01:33:05.855 sometimes to go through these time lapses because we kind of want to see what happens in one area in the experiment. So here you have something 01:33:05.875 --> 01:33:15.965 that can let you listen through it, but also you can kind of flip through it and see if you maybe hear something weird. This is kind of similar to 01:33:16.505 --> 01:33:28.077 you know, our vision in real life. When you hear something interesting you turn your head towards it and try to look at it. So I guess this kind of 01:33:28.137 --> 01:33:40.347 sums up the whole thing and again my approach here or my motivation really is to create an audio-visual real-time environment that can 01:33:40.407 --> 01:33:54.302 hopefully plug into your whatever analysis workflow there is and hopefully lets you explore your initiata in a fun and hopefully 01:33:54.322 --> 01:33:55.003 interesting way.