Biology | 2i2c

Introducing the Catalyst Project Community Partner Highlights

Thu, 31 Oct 2024 00:00:00 +0000

Cross-posted from the Catalyst Project blog

Catalyst Project community partners using accessible cloud infrastructure for open science leadership and training. (clockwise from top-left) NNB-CCG, MUST, CICADA and INER. Photos courtesy of Shirley Alquicira Hernández, Bennett Kankuzi, María Inés Fariello Rico and Yalbi I. Balderas-Martinez.

The Catalyst Project is a community-engaged initiative designed to support the adoption of open science principles in under-served bioscientific research communities through the provision of reliable and sustainable cloud computing infrastructure. It’s a project we’ve been working on now for almost two years, which involves staff from seven different organizations: 2i2c, The Carpentries, CCAD, CSCCE, IOI, MetaDocencia, and OLS, and is funded by the Chan Zuckerberg Initiative.

A key part of the project is engaging with Community Partners in Africa and Latin America: Institutions, organizations, and individuals who are undertaking bioscientific research projects that require cloud computing infrastructure. As collaborators on the Catalyst Project, Community Partners can access and use 2i2c’s open science cloud services, and also receive training from 2i2c, The Carpentries, MetaDocencia, and OLS to support their work. Community Partners also play a vital role in shaping an evolving governance model for the Catalyst Project to help sustain, scale, and maximize impact in Latin America, Africa, and under-served communities around the world.

In a new collection of blog posts (that we hope will expand over the next couple of months!) we’re highlighting the work of the Catalyst Project Community Partners. This post is a gateway to learning more about the Catalyst Project and its Community Partners. If you have any questions or feedback about the project, please send an email to the core team.

Highlighting the Catalyst Project Community Partners #

The Catalyst Project currently involves 19 Community Partners, 9 in Africa and 10 in Latin America. Our initial blog post series showcases seven of the Partners, and each post is available in English and Spanish:

African Institute of Biomedical Science and Technology (AiBST) #

At the African Institute of Biomedical Sciences and Technology (AiBST) in Zimbabwe, Zedias Chikwambi and colleagues are working to discover and utilise biomarkers to guide personalized medical treatment.

“Through the Catalyst Project we are able to bring genomic information interpretation to patient bedsides.” - Zedias Chikwambi

Read more: EN | ES

Malawi University of Science and Technology (MUST) #

The Catalyst Project Community at the Malawi University of Science and Technology (MUST) is working to popularize the applicability of data science and artificial intelligence in various undergraduate and postgraduate research contexts in Malawi, including health, business, and education.

“…many staff and students need… a robust and easily accessible platform from which they can efficiently run their machine learning models and do advanced data analysis for their data science research…” - Bennett Kankuzi

Read more: EN | ES

MolerHealth #

MolerHealth is focused on revolutionizing healthcare in Nigeria by developing an accessible electronic health record (EHR) system aimed at reducing disease misdiagnosis.

“Access to training, like The Carpentries Instructor Training, has empowered our team with essential skills for effective teaching and collaboration.” - Monsurat Onabajo

Read more: EN | ES

Nelson Mandela African Institution of Science and Technology (NM-AIST) #

The Northern Tanzania One Health Research Group, hosted at the Nelson Mandela African Institution of Science and Technology (NM-AIST), is using the Catalyst Cloud Infrastructure to understand the transmission dynamics, genetic diversity, and antimicrobial resistance of the Mycobacterium tuberculosis complex (Mtb) between humans and livestock in Northern Tanzania.

“…access to training, particularly through the 2i2c Hub Champion Training, has significantly enhanced our ability to manage and optimize cloud-based resources." - Beatus M Lyimo

Read more: EN | ES

Nodo Nacional de Bioinformática (NNB-CCG) #

The Nodo Nacional de Bioinformática (NNB-CCG) of the Centro de Ciencias Genómicas (CCG) - Universidad Nacional Autónoma de México (UNAM) is a group that brings together professionals and academics to support, provide services, and maintain the growth of the field of bioinformatics in the country’s research.

“(Our) goal is to optimize our participation in events, assess the usefulness of the Catalyst Project’s resources, and, in turn, provide the Catalyst Project with guidelines to improve their service by identifying the necessary areas for improvement within the institutions." - Shirley Alquicira Hernández

Read more: EN | ES

Instituto Nacional de Enfermedades Respiratorias (INER) #

Collaborators at INER are using the Catalyst Cloud Infrastructure to implement machine learning algorithms that will classify radiology images of pulmonary thromboembolism (PTE), with a view to improving patient outcomes.

“The Catalyst Project is helping us to collaborate more efficiently and work remotely.” - Yalbi I. Balderas-Martinez

Read more: EN | ES

Centro Interdisciplinario en Ciencia de Datos y Aprendizaje Automático (CICADA) #

CICADA, an interdisciplinary center researching data science and machine learning, is using the Catalyst CLoud Infrastructure to analyze Uruguayan population data to understand patterns of migration, how much of the native footprint remains, and what can be said about the people who lived in the Uruguayan territory before the arrival of Europeans.

“The Catalyst Project…trainings are attractive, as they are respectful of the people, no previous knowledge is assumed, and the instructors are welcoming.” - María Inés Fariello Rico”

Read more: EN | ES

About the blog posts – Acknowledgements #

Creating a series of blog posts to highlight the work of the Catalyst Community Partners was a collaborative effort involving staff from CSCCE, 2i2c, OLS, and MetaDocencia. Specifically, Lou Woodley, Katie Pratt, Jenny Wong, and Tajuddeen Gwadabe conceived the idea during our regular Catalyst Project “website team” meetings, and developed a strategy for reaching out to community partners to gather information.

Katie then led that outreach, which involved a representative of each Community Partners completing a Google Form to answer three key questions and sharing a photo (or a selection of photos) to use on the blog. Katie curated all of the information, edited together the blog posts, and secured sign-off from all of the partners. Sabrina López and her team at MetaDocencia coordinated the translation of the posts from English to Spanish, and Jenny formatted and published them on the website.

Acknowledgements #

Thanks to the Catalyst Community Partners for generously sharing more about their work.
Thanks to CZI for funding this work.

Hub Champion Training Reflections (English)

Wed, 04 Sep 2024 00:00:00 +0000

We’ve developed and are now offering a Communtiy Hub Champion training, see this blog post from the Catalyst Project for more details.

Acknowledgements #

Thanks to the Catalyst Project for this work.
Thanks to CZI for funding this work.

Reflexiones sobre la formación de Campeones y Campeonas del Hub (Español)

Wed, 04 Sep 2024 00:00:00 +0000

MyST Mini-Hackathon with the DeepLabCut Team

Mon, 02 Sep 2024 00:00:00 +0000

The DeepLabCut Team #

Animal pose estimation using deep neural networks. Courtesy of the DeepLabCut Jupyter Book

The DeepLabCut team is a group of researchers and developers who are working on open source tools for analyzing animal pose estimation by training deep neural networks on videos.

Chris Holdgraf visited the lab in early August to learn more about how the group were using open-source tools to document and share their work.

Jupyter Book and MyST #

Extensive documentation for using the DeepLabCut software package is already available as a Jupyter Book. The group was interested in adopting MyST Markdown to stay ahead of the curve and upgrade their Jupyter Book (see the related announcement Jupyter Book 2 will be build upon the MyST-MD engine).

Chris led a mini-hackathon to introduce the group to MyST and collect feedback on where enhancement features could be made in the future. Here’s a summary of the outcomes:

Many improvements were made to the MyST documentation 📖
- The MyST Quick Start Guide was used to onboard new users. Amendments were upstreamed to the MyST docs directly and were immediately available to all.
- A tutorial on executable documents was added to the collection of MyST tutorials.
- MyST-MD installation instructions were simplified using mamba.
A bunch of enhancement features were requested ✨
And we found a bug in the table of contents validation 🐞

Summary #

Hackathons are a great way for quickly imparting knowledge and gathering feedback in a short space of time. The event spurred rapid contributions to the MyST ecosystem – embracing reuse of the MyST quick start guides saved time and effort, while engaging with users directly closed a tight feedback loop for enhancements.

Acknowledgments #

Thank to the Mackenzie Mathis Lab for hosting Chris Holdgraf at EPFL, Lausanne, Switzerland.
Thanks to the Jupyter Book team for collaborating on this with us.

NeuroHackademy Summer School Reflections

Mon, 19 Aug 2024 00:00:00 +0000

Thank you to Ariel Rokem and Noah Benson for guest writing this blog post!

Group photo from NeuroHackademy 2024

What is NeuroHackademy? #

Part summer school, part free-wheeling hackathon, all focused on the use of data science methods in neuroscience, NeuroHackademy is an event that was recently hosted by the University of Washington eScience Institute in Seattle, WA, USA. This event, that has been running annually since 2016, aims to provide early-career researchers in Psychology, Medicine, Neuroscience, and other related fields with the skills and knowledge that they need to effectively and rigorously work with open source tools and workflows for analyzing human neuroscience data. This supports the effort to make scientific analysis and results shareable, reproducible, and accessible.

Global and inclusive #

In 2020, the event had to rapidly pivot to an online format, and this format was also used in 2021. Through this experience, the organizers ( Ariel Rokem and Noah Benson) realized that many participants preferred the online format. For example, participants who could not afford to travel to Seattle, or participants who had care-taking responsibilities that precluded them from participating in a two-week event away from their homes. In 2022, the event pioneered a hybrid format, where half of the participants are present in-person and half join the event via zoom, slack, GitHub, and of course through a dedicated 2i2c JupyterHub. Taken together, this format allows the participation of students from a larger range of backgrounds and locations. This aspect plays an important part in building a global and inclusive community of practice. See the paper Hands-On Neuroinformatics Education at the Crossroads of Online and In-Person: Lessons Learned from NeuroHackademy to read more on this subject.

Collaboration with 2i2c #

Previous years #

NeuroHackademy has been an early adopter of the cloud-based JupyterHub model, setting up its first hub using the zero-to-jupyterhub guide in 2018. NeuroHackademy partnered with 2i2c as soon as it was founded, and 2i2c has operated a JupyterHub for the last 3 years. The hub provides an interactive computing platform for learners, and implements the “digital watering hole” for practical and immediate access to a range of cloud-based datasets in human neuroscience (see blog post announcing support for this year’s event).

In terms of the software environment, the following tools and features that have benefited the event over the years include

nbgitpuller allows students to synchronise lesson content with an organizational GitHub repository that is collaboratively maintained by the lesson instructors.
Shared data file storage with read-only access for learners and read-write access for instructors
Access to an abundance of neuroimaging data hosted in cloud object storage
- The Human Connectome Project
- The Natural Scenes Dataset
- OpenNeuro
- The Healthy Brain Network
- And more.

This year #

This year 2i2c supported the following tools and features for NeuroHackademy

A “Bring your own image” option where users can pull any image hosted on a container registry into the hub. See our Integrating BinderHub with JupyterHub: Empowering users to manage their own environments blog post for more details.
repo2docker and GitHub actions to build and prototype images from a repository.
The support services provided by 2i2c and the ability for instructors to open pull requests on 2i2c infrastructure for speedy resolution.
GPU instances to support more compute intensive workloads for machine learning.

Next year #

One thing we have learned is that 2i2c automatically shuts down a user server after one hour of inactivity by default to ensure efficient resource usage and limit runaway cloud costs. Naturally, we are seeing increasing demand from learners for longer and more complex analyses. In response to this, we are keen to explore how the jupyter-keepalive extension can keep the server alive for long-running processes.

We are pleased that learners have made great progress in making use of cloud-native, open-source workflows for analyzing human neuroscience data. We are keen to benefit from lessons learned this year and are looking forward to collaborating with 2i2c once again to deliver the NeuroHackademy Summer School in 2025.

Watch this space next year!

Acknowledgements #

Funded by grant R25MH112480 from the US National Institute of Mental Health awarded to Ariel Rokem and Noah Benson.

The NeuroHackademy Summer School is sponsored by

References #

Enabling neuroscience in the cloud with HHMI Spyglass and MySQL on JupyterHub

Fri, 05 Jul 2024 00:00:00 +0000

The HHMI Spyglass tutorial

Spyglass #

Spyglass is a framework for reproducible and shareable neuroscience research produced by Loren Frank’s lab at the University of California, San Francisco. Check out our blog post about the release of their preprint to read more about the methods.

This post focuses on the complex data storage needed for the project, which can be difficult to set up locally or at scale in the cloud. In particular, the analysis needed a MySQL database for reproducibility. This is a fairly common task across many fields. The aim of 2i2c is to enable researchers to focus on the essential complexity of what they were doing, i.e. the science, without managing the accidental complexity of how to do it – in this case, setting up databases.

We describe how you can do this too for your own JupyterHubs. Since 2i2c commits to running our infrastructure in line with open-source values as much as possible, you can also directly see the configuration for the hub referenced in the paper.

What is a “sidecar container”? #

The Kubernetes definition of a sidecar container is

Sidecar containers are the secondary containers that run along with the main application container within the same Pod. These containers are used to enhance or to extend the functionality of the primary app container by providing additional services, or functionality such as logging, monitoring, security, or data synchronization, without directly altering the primary application code.

In this case, the primary app container is the JupyterLab instance where people are interactively running code and doing science. We want to provide a MySQL database as a sidecar so that each user server gets their own independent MySQL server instance (that is not accessible to anyone else). We can then run code such as

%%bash
mysql -h 127.0.0.1 -u root --password=tutorial < path-to-sql-file-with-data

to load data into the database. Note the IP address 127.0.0.1 - the MySQL server is listening on localhost, even though it is not running in the same container! Thanks to the magic of Linux Network Namespaces, the sidecar and main app container can share 127.0.0.1. This allows you to write code that works in the exact same way on a user’s local computers as on the JupyterHub, making transitions and replication easier.

Setting up sidecars in JupyterHub on Kubernetes #

We’re leveraging multiple tools from the open-source ecosystem - JupyterHub, Kubernetes, Linux as well as MySQL itself.

Since this is a Kubernetes feature, we can pass through config to it. There are two layers here, which are

singleuser.extraContainers in z2jh configuration
KubeSpawner.extra_containers in KubeSpawner configuration

The hub configuration looks like

 singleuser:
 extraContainers:
 - name: mysql
 image: datajoint/mysql:8.0 # following the spyglass tutorial at https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/#existing-database
 ports:
 - name: mysql
 containerPort: 3306
 resources:
 limits:
 # Best effort only. No more than 1 CPU, and if mysql uses more than 4G, restart it
 memory: 4Gi
 cpu: 1.0
 requests:
 # If we don't set requests, k8s sets requests == limits!
 # So we set something tiny
 memory: 64Mi
 cpu: 0.01
 env:
 # Configured using the env vars documented in https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/#existing-database
 - name: MYSQL_ROOT_PASSWORD
 value: "tutorial"

By setting this up, we allow users to insert the code snippet above

%%bash
mysql -h 127.0.0.1 -u root --password=tutorial < path-to-sql-file-with-data

into their Jupyter Notebooks, which gives access to their MySQL database in the hub!

However, this configuration does not include permanently store the database itself between hub server sessions. Thanks to a pilot in a prior collaboration with University of Texas, Austin, we do have some documentation on how you can enable that as well!

Acknowledgements #

Howard Hughes Medical Institute
National Institute of Mental Health (NIMH), grant number RF1MH130623
kubespawner
zero-to-jupyterhub-k8s and the JupyterHub community

Neurohackademy Summer School in Neuroimaging and Data Science 2024

Fri, 21 Jun 2024 00:00:00 +0000

Neurohackademy Summer School

2i2c are pleased to support the Neurohackademy Summer School in neuroimaging and data science again!

Following the success of our collaboration in previous years (see the event page for 2023), this year’s course will be held July 29th – August 10th, 2024 and will be hosted by the University of Washington eScience Institute.

We provide an interactive computing platform for participants to get hands on experience in data pipelining, machine learning and data visualization techniques. Take a look at the following links to learn more about the neurohackathon:

Acknowledgements #

Funded by grant R25MH112480 from the US National Institute of Mental Health awarded to Ariel Rokem and Noah Benson.

The Neurohackademy Summer School is sponsored by

Howard Hughes Medical Institute publishes preprint on Spyglass, a framework for reproducible and shareable neuroscience research

Tue, 21 May 2024 00:00:00 +0000

Spyglass landing page

Spyglass is a framework for reproducible and shareable neuroscience research produced by Loren Frank’s lab at the University of California, San Francisco. They recently released a preprint about their toolbox, and are using a 2i2c hub to provide accessible interactive cloud environments that demonstrate its functionality and helps researchers get started.

Acknowledgements #

Thanks to HHMI for funding and collaborating on this work.

UNITEFA forms the first community of the Catalyst Project at UNC, through the CCAD (in Spanish)

Mon, 23 Oct 2023 00:00:00 +0000

New project: Open science cloud infrastructure and training for communities in Latin America and Africa

Tue, 20 Dec 2022 00:00:00 +0000

We are excited to announce that the team and proposal described in this blog post has been awarded funding by the Chan Zuckerberg Initiative! We informally call this project The Catalyst Project.

This announcement may be cross-posted on the websites of several collaborating organizations of this grant. Para leer este post en español, vea el blog de MetaDocencia.

Our goal is to create a collaborative cloud infrastructure service that enables community-based cloud-native workflows in the biosciences. We will promote values of open and inclusive community practices, infrastructure that enables these practices, and a “train the trainers” approach that empowers community leaders to share expertise in cloud infrastructure with others in their communities. Our focus will be on communities in Latin America and Africa, and we hope to learn how this model could be extended to other global communities that are historically marginalized from large-scale scientific infrastructure projects.

2i2c will be providing cloud infrastructure operation and support for the communities that we partner with in this effort. We will also assist with creating content to teach cloud-native workflows and assist community leaders in learning this content so that they can share these skills with others.

This is a collaborative effort between 2i2c, The Carpentries, CSCCE, Invest in Open Infrastructure, MetaDocencia, and Open Life Science. For more detailed information, see the blog post with our full grant narrative.

We are hiring #

As a part of this effort, we will also hire several new team members! There are currently two job postings open. Here are links for more information in case you are interested:

Cloud infrastructure engineer to join 2i2c’s Site Reliability Engineering team that will operate and support the cloud infrastructure in this project.
Programme manager role to join Open Life Science and support this project via project management and operational support.

We may be hiring other positions related to this effort, so please stay tuned for more information if you are interested.

Where to follow along #

If you’d like to follow along with this work, please share your e-mail address in this short form. We’ll send updates as we work out longer-term spaces for communication or documentation.

💡 Follow our work! Sign up for our mailing list for updates about 2i2c. Send us an e-mail about collaborating or partnering on a project. See our Service Documentation or our Team Compass to learn about our service and organization.

Grant progress report: CZI Foundational grant year 2

Sun, 13 Nov 2022 00:00:00 +0000

We recently completed a progress report for Year 2 of our primary CZI funding grant. This funding covers some core operations of 2i2c as well as engineering capacity to run our cloud infrastructure for JupyterHubs.

Below is a link to the 3-page grant narrative that summarizes some of our major progress and milestones from year two:

zenodo.org/record/7319289

Acknowledgements #

Thanks to CZI for funding this work.

Open grant narrative: A Collaborative Interactive Computing Service Model for Global Communities

Sun, 28 Aug 2022 00:00:00 +0000

We recently submitted a grant to Chan Zuckerberg Initiative and wish to share some details about it as well as the grant narrative for others to read and re-use.

Go to Zenodo record

Read on for a quick overview of the proposal.

Collaborators #

This grant is a collaboration between several leading organizations in open infrastructure, community, and global leadership:

Problem statement #

Cloud infrastructure is a powerful way to broaden access to workflows and infrastructure across the globe. However, it is also inaccessible to many for a variety of reasons:

There is a large, diverse, and messy ecosystem of open source tools to facilitate cloud infrastructure.
Most communities don’t already have skills in utilizing cloud workflows.
Running infrastructure in the cloud takes dedicated time and expertise that many communities lack.
Many communities do not have organized communities of practice around cloud infrastructure.

These issues are true for most scientific communities, but they are exacerbated in countries that are often marginalized in the global scientific community.

Our proposed work #

For this reason, our goal in this grant is to provide human and technical services to facilitate learning and knowledge transfer of cloud workflows for communities in Latin America and Africa.

It defines four major areas of collaboration:

Cloud infrastructure management - to facilitate access to cloud resources via managed cloud services that integrate open source tools.
Application guidance and training - to provide community leaders with the skills to utilize this infrastructure for their needs. This includes language-localized content and training materials.
Training for trainers - to provide community leaders with skills to share these workflows with others in their communities.
Community leadership and management - to provide community leaders with skills to sustain and grow healthy communities of practice.

If this proposal is funded, over the course of two years the team will provide a combination of the services described above for communities in Latin America and Africa, with the goal of understanding how such services can be most useful for these communities, how we can structure them to provide community representation in the direction of these services, and how we can sustain and scale this model of community-focused services for a global community.

Importantly, we wish to do this work in a way that centers the communities we work with as co-leaders and collaborators in these services. We will explore ways to run these services and workshops so that they are transparent, inclusive, and give agency to the communities they support. Ultimately, we hope that this can be an extensible model for many more communities in the future.

Acknowledgements #

Thanks to CZI for the opportunity to submit this proposal.