.png)
Course description
AI agents stand ready to take on time-consuming data wrangling tasks from researchers, such as reformatting, cleaning, and integrating data from different sources. “Vibing” dataflows into existence is now easier than ever. However, your typical spaghetti-coding PhD-student has more than met their match as AI agents push the limits for how unreadable and unmaintainable research code can really become.
And the data? What really happens to your valuable research data through transportation and transformation? Well, ask your agent, and they’ll say: “It’s all good! Trust me, Bro!”
Or can we do better? Omnipy is an UiO-developed Python library that offers a systematic and scalable approach to building data pipelines. The design is solid and hand-made – development has been going on since before ChatGPT was unveiled to the world. Omnipy was made for humans, to simplify development, deployment and maintenance of dataflows. Building on Pydantic and Prefect, two hugely powerful Python libraries, it offers a declarative syntax for pipeline creation – what goes in, how the input looks like, how the output looks like, which pipe goes where, how they fit together – and data fetching, validation, conversion, and staging largely comes out by itself. Omnipy has, however, come with a threshold: new abstractions take some effort to wrap your minds around!
With the rise of AI agents, it has become clear where the real power of Omnipy lies: as a middle-ground for humans and machines. Omnipy is your companion from start to finish, from fetching and inspecting the data to large-scale deployment of a data pipeline. It provides tools for URL building, paging, and asynchronous fetching of data – then full-screen, syntax-colored inspection of any data type: bytes, text, hierarchical or tabular; in the terminal, in Jupyter, or in the browser. You follow up with some data extraction, add some filters, perhaps flatten the JSON to a Pandas table, clarify the types, and build a plan.
It’s time to bring in the cavalry. The AI agent learns the syntax and turns your hand-written exploration code into clearly defined data models, pipes and conversions, while you inspect it all, tighten the knobs, stop all ridiculousness. Then you pour in some test data, let Omnipy and Pydantic parse and validate the data at each turn; the agent fixes the leaks while you inspect the data in full detail at all the junctions. The declarative dataflow syntax functions as a readable contract between you and your agent – everything is in your control. In the end, you scale up, push the dataflow onto remote pipeline deployment with industry-scale orchestration management, and open the floodgates!
Building from experience from earlier DSDays workshops, we are changing the format. This workshop is for early adopters and medium-to-power users. The aim is to present the gist of the functionality and possibilities on real-world examples rather than step-by-step tutorials. We want to build collaborations and open up for contributions. The modularity of Omnipy, with its power to wrap any other data structure in Python, from any library, and built for reusable pipeline pieces, allows for integrations in many directions. Example integrations will be presented, such as RO-Crate and BioNumpy.
It is time to expand the scope from in-house scenarios to also cater for your use case, requirements and competence. In short: it’s time to start a community of pipeline builders!
There will be a half-day, intensive workshop before lunch. After lunch, we will open for discussions, idea-sharing, and collaboration building. We will start in a group setting, then move into focused slots, for discussions of particular collaborations, themes, demonstrations and follow-ups, one-to-one or in groups, as needed (details will be provided later). Attendance to this part will be flexible.
Target audience
Technically-oriented individuals of every relevant position, master students to PIs, engineers / IT staff and anyone interested across scientific domains.
Learning outcomes
- Understanding of the various tools, functionalities and modalities of Omnipy:
- URL building and robust asynchronous data fetching, with rate limiting and retries.
- Data visualization, pluggable dynamic pretty-printers, syntax coloring and over 300 color styles!
- How Pydantic-based data modeling in line with the “Parse, don’t validate” idea allows for flexible data parsing, transformation and validation.
- How robustness is built into every piece, with atomic snapshots and dynamic validation.
- Omnipy Model mimicking of any Python class from any library
- How Dataset and type specificity allows for large-scale structuring of data for direct batch processing, filtering, and operations without for-loops.
- Working with hierarchical data, nested datasets, flattening to tables.
- Working with tabular data, conversions (row-based ⇔ column-based, header definitions, etc), and type-based validation across libraries.
- Different types of data flows: linear, DAG, and functional.
- Job modifiers as high-level knob turning.
- Engine backends: local or Prefect (with possibilities for extensions).
- Integration examples: RO-Crate and/or BioNumpy.
- How all of this allows for a new way of working with AI agents, with explicit contracts and direct data inspection, ending up with maintainable and scalable data pipelines for FAIR and reproducible research.
- Discussions of collaboration/integration possibilities.
Prerequisites
- This workshop is for medium-to-advanced Python developers / AI agent-powered builders, technically oriented researchers with data pipelining needs, and PIs providing potential collaboration projects and use cases.
- Medium-to-advanced Python knowledge is assumed, however technically-oriented PIs/managers will probably still find the workshop useful for grasping the high-level functionalities without having to understand every detail.
Required materials
- Laptop needed to follow presented use cases hands-on (JupyterLab installation will be provided).
- A form for signing up for dedicated slots or suggesting topics for the after-lunch session will be made available prior to the workshop and will remain open until over lunch on the day.
Lunch
- There will be a complementary buffet-lunch at 12:00-13:00 from the Sverdrups spiseri (Georg Sverdrups Hus, 2nd floor).
- After you get your lunch, join us at Store møterom where we have reserved seats for #DSDays2027 participants.
Number of participants
25
Contact
If you have any questions regarding this course, or anything else about the DSDays 2027 event, please contact us per email: digitalscholarship@ub.uio.no