Please note!
This half-day workshop is Part 1 of a two-part series, providing the essential skills for the intermediate-level Part 2.
In Part 2, participants will learn to define and orchestrating data flows, including integrating with data sources and deploying flows onto external compute resources.
You may attend Part 1 alone or join both sessions for a complete experience; however, attending Part 1 is recommended as it lays the groundwork for the more advanced topics in Part 2.
Course description
Researchers often spend a significant amount of time on data wrangling tasks, such as reformatting, cleaning, and integrating data from different sources. Despite the availability of software tools, they often end up with difficult to reuse workflows that require manual steps.
Omnipy is a new Python library that offers a systematic and scalable approach to building data pipelines. Through specification of data models/parsers, Omnipy allows researchers to import data in various formats and wrangle their data through stepwise transformations.
For automation with large data, Omnipy seamlessly scales up for deployment on remote infrastructures. This workshop will provide down-to-earth tutorials and examples to help data scientists from any field make use of Omnipy to wrangle real-world datasets into shape.
The workshop is divided into three parts:
- The first part will introduce the concepts of models, datasets and tasks in Omnipy through small examples. We will also touch upon Python type hints and pydantic models as needed, as these are important building blocks for Omnipy.
- In the second part, the participants will be provided with a rough example dataset that require cleaning. As a hands-on exercise, the participant will carry out step-wise parsing and shaping of the data to make it comply with a specified metadata schema.
- In the last part, the participants will be introduced to the metadata mapping functionalities in Omnipy and will be led through another hands-on exercise to set up a transformation that maps the data from one metadata schema to another.
Target audience
PhDs, postdocs, and technical personnel with academic programming experience. Data science will be a particular focus, but the workshop is open to any interested participants.
Learning outcomes
- Introduction to Python type hints and pydantic models
- How to use type hints to define models, datasets and tasks in Omnipy
- How to wrangle a rough dataset into the shape required by a metadata schema
- How to set up an executable mapping of data from one metadata schema to another
Prerequisites and required materials
- The participant should have some experience with Python programming/scripting. We will not spend time explaining basic syntax and concepts, other than what is related to type hints. Experience with type hints in Python is useful, but not required.
- Laptop.
- No software installation is required other than a modern browser.
- We will make use of JupyterLab for the hands-on exercise. An online JupyterLab service will be made available, but participants can also install JupyterLab locally on their laptop if they prefer.
Lunch
- There will be a complementary buffet-lunch at 12:00-13:00 from the Sverdrups spiseri (Georg Sverdrups Hus, 2nd floor).
- After you get your lunch, join us at Store møterom where we have reserved seats for #DSDays25 participants.
Number of participants
After the soft deadline, a waiting list will be automatically handled upon de-registrations.
Contact
If you have any questions regarding this course, or anything else about the DSays25 event, please contact us per email: digitalscholarship@ub.uio.no
Sign up for this workshop!