Report | 2i2c

Report from the Jupyter Security Working Group security tooling sprint

Wed, 08 Apr 2026 00:00:00 +0000

The Jupyter Security Working Group recently held a Security Tooling Sprint. It was a timely event given the recent spate of software supply chain attacks across the tech world.

The sprint covered two main areas:

Governance and strategy — conversations about responsibility and accountability in the face of AI, with emphasis on ensuring humans are ultimately responsible for code committed to Jupyter subprojects. The group also discussed how security could benefit from working group members regularly attending subproject meetings like the JupyterHub Collaboration Cafes.
Automation and tools — the group evaluated several tools for improving security posture across the Jupyter ecosystem. Here are a few that stood out:
- Semgrep as an alternative vulnerability scanner to CodeQL
- Grype, Checkov, and Kubescape for cloud infrastructure misconfiguration checks
- Schemathesis and restler-fuzzer for API fuzz testing

One challenge we discussed was how blindly running security scanning tools generates many false positives. There’s real effort needed to tune these tools for each project’s edge cases before they’re useful in automation. On a related note, we discussed the increase in AI-generated (or AI-assisted) vulnerability and security reports, and the challenges associated with sifting through all of those pieces of information.

Acknowledgements #

Thanks to the jupyter security working group for providing leadership and organizing, in particular Joe Lucas!
Thanks to the Jupyter Foundation for funding community meetings like these.

2024 impact report: new team structure, new funding, and growth in our network

Sun, 08 Dec 2024 00:00:00 +0000

2024 has been a busy year for 2i2c, with many highs and lows, a lot of impact, and significant organizational change. As the year comes to an end, I want to reflect on the work we’ve done in 2024, and where we aim to go in 2025.

In 2024, 2i2c reached the point in an organization’s lifecycle when a team has grown enough in size and complexity that you must change the ways that you organize. The informal ways that worked as a small group don’t suffice anymore, and you have to put more effort into aligning and coordinating everyone to ensure you have the same impact.

I call this the “$1M to $2M budget jump”, because organizations seem to hit this point around when your annual budget goes from $1M to $2M. Getting to the other side of this gap with an intact runway and team is hard, and I suspect that 2i2c’s fully distributed nature means that we hit these scaling milestones earlier than many organizations. For us, this has been a major focus of effort all year, and has involved taking a top-to-bottom look at our plans and ways of working as a team. Read on for more details about major updates, challenges, and impact that our team had in 2024.

Organizational updates #

At an organizational level, this year had a lot of introspection and planning, a few new roles, a few departing team members, a funding crunch, a successful effort to dig out of it, and a new system of work organizing our team. We’ll share more about all of this later, but here are the major implications for our team:

We’ve raised another $2.2M in funding to support our efforts in scaling and sustaining our network of community hubs. This gives us roughly another 2 years of projected runway (with some assumptions about revenue from contracts and grants). Below are two posts that describe two major awards we were awarded in Q3 and Q4 of this year:

We’ve designed and hired several strategic and systems-level roles to give our team support and direction as it grows. Here’s a brief summary:

Head of Product, Giuliano Maciocci. Giuliano leads our efforts to define and steward our value proposition and the roadmap of development for products and services that feeds into it. Giuliano has 10+ years experience driving growth in companies, was the ex-Chief Product Officer at Ex Ordo, ex-head of Product at eLife Sciences where he led the open science product Executable Research Article, and contributed significantly to mobile design and innovation at Adobe.
Chief of Staff and Delivery Manager, Harold Campbell. Harold leads our delivery and operations efforts, and stewards our system of work and coordination around it to ensure we deliver on our commitments efficiently and reliably. Harold has 15+ years of industry experience spanning companies in Africa and Jamaica, and 10+ years experience in agile consulting and coaching in technical and product teams.
People Lead, April Johnson. April leads and stewards our system to support our team as individuals, ensuring that we provide the guidance and support needed to grow our team members in their careers and skills. April has 20+ years growing and guiding technology organizations. She is the ex-Global Head of Transformation at Thoughtworks (Europe, Latin America, India, North America, and Asia), with expertise in human-centered design, agility, people leadership, change, leading remote and asynchronous agile teams, coaching, and non-profit development.

We’ve re-organized our team into separate product and business development teams, in order to focus on providing an excellent technical platform and a collection of services that maximizes community impact, as well as sustaining this service for our communities. This has allowed us to more effectively coordinate our service enhancement and development efforts, and increases our ability to deliver improvements to our communities and to upstream projects.

As a result, our organization has a much stronger foundation to build upon as we continue to grow and refine our sustainability model in 2025. It has positioned us to more effectively deal with the challenges in reaching our next milestones for scale and impact, and gives us the tools to be more adaptive and responsive to community needs.

Along the way, we also generated a lot of impact through our collaborations with communities in our network, and in the upstream projects that we support. For more details about our impact, see the summaries below.

Community impact #

2i2c’s core mission is to support its network of communities that create and share knowledge with open infrastructure. Here are the highlights of how our community network has grown and had impact.

First, we’ve grown our network of hubs and users through several new partnerships. We grew the number of active hubs from ~75 to ~105, and grew our end-of-year Monthly Active Users (MAUs) from ~6000 to ~8000.

You can see an interactive version of these numbers in our platform usage dashboard: 2i2c.org/kpis/cloud/

Beyond the numbers, we also re-focused our team on reporting impact stories from our collaborations with community members, and have published these into a (growing) list of posts on our blog:

Here are a few community highlights from this year:

We served around 20 communities from Latin America and Africa for the Catalyst project
Our community partner Openscapes were invited to the White House to discuss the importance of open science
The NeuroHackademy used our infrastructure to support their annual summer school
We enabled ephemeral and sharable interactive computing environments for the Amerigeo workshop in the geospatial community
We ran a pilot for an HHMI-affiliated open source project called Spyglass for reproducing their pre-print with a live interactive environment using BinderHub to support publishing infrastructure
We began a collaboration with Development Seed around the NASA VEDA project, to support them with interactive cloud environments for geospatial research
Several members in our community network showed off their work at AGU 2024
We co-organized a workshop alongside Project Pythia to create computational narrative content for geospatial analytics, and upgrade their stack to Jupyter Book 2.0

Open source technology enhancements #

Our second pillar of impact is to improve the ecosystem of open infrastructure and the open science workflows it enables. We use collaborations with our community partners to drive new cycles of development in open source tools that we support. Here’s a brief overview of our impact across the open source ecosystem this year.

In 2024, 2i2c team members authored over 500 pull requests that were merged in our key open source communities communities. Find our list of key open source communities here:

compass.2i2c.org/open-source/key-communities/

Here are a few highlights where we focused our effort this year - each of these efforts required both development with and for our community network, as well as upstream contributions and support:

We released JupyterHub Fancy Profiles, which allows for a more flexible and modern interface to launch environments with JupyterHub.
We used this to allow users to build and launch custom environments in JupyterHub in a way that users can also share with others.
We’ve added a Grafana dashboard for resource and cost monitoring with JupyterHub to give communities more visibility over their projected cloud costs.
We began incorporating Jupyter Book 2.0 workflows into our community hubs and laid a foundation for enabling our communuty networks to communicate with one another more effectively using the new MyST document engine. Read more in our blog post about Jupyter Book 2.0.
We built frx-challenges, a tool to help communities host data challenges with secure, automated evaluation of submissions. This was built in collaboration with the HHMI Cellmap Challenge competition.

Looking to next year #

2025 is going to be a critical year for 2i2c to build upon the work we began in 2024 to achieve a more sustainable and scalable community model. Here are the main areas that will guide our work in the new year and into 2026, pulled from our recent proposal from The Navigation Fund:

Goal #1: Delivery. Develop the operating structure and team skills to efficiently scale our product and service delivery.
Goal #2: Product. Develop a product system that continuously improves and delivers value and impact at scale.
Goal #3: Sustainability. Build a business model that is competitive and gives us resources to sustain and scale our service.

These are the key goals 2i2c must achieve in order to ensure that its service remains impactful, sustainable, scalable, and accessible. We believe that we’ve laid a strong foundation to get there, and are excited to begin work next year.

Overall, 2024 has been a challenging, but also a rewarding year for our team. We’ve encountered and successfully worked thorugh a number of scaling challenges, and we’ve made significant progress at laying a foundation on which we can build for the years to come.

I’m incredibly proud of 2i2c’s team for all of their hard work this year, and also honored to be working with a network of communities that care about open infrastructure and its value for creating and sharing knowledge with the world. Here’s to another year of impact!

Acknowledgments #

Support for organizational and strategic work like this is provided by a grant from The Navigation Fund and CZI.

Community Update: October 2023

Fri, 10 Nov 2023 00:00:00 +0000

Over the past few months, we’ve been investigating ways to improve our reporting both internally and externally. We’ve decided to experiment with a monthly community update to create a regular cadence of transparency and highlights from 2i2c for our broader community. This is the first such update, so bear with us as we work out the kinks!

Our goal with these updates is to share what stands out at 2i2c - what we’ve learned, what we’re proud of, where we’ve struggled, and where we’ve had an impact. We hope this can be a historical record of “what stands out” to our team and that it is useful for our broader community to see. We also want this to be relatively short and to the point to make it sustainable to both write and read.

We’d love feedback on what else you’d like to see. If you have any ideas, please send an email to hello@2i2c.org.

Rough Numbers #

First off, a few numbers on the scale and status of our interactive computing hub service:

We are currently running around 73 JupyterHubs that are collectively averaging more than 4000 average monthly users. We have several communities in the educational sector, so October was a peak in the Fall semester of activity.

We’re recovering roughly 40% of our monthly staffing costs through recurring service revenue. The remainder we’re making up with a combination of grants and more focused development contracts.

Assuming minimal growth in our service and fundraising, our runway is roughly until August 2025 - however, we anticipate this to shrink in the short-term once we make critical new hires in the coming months.

Organization Updates #

This section describes some major organization-wide efforts we’ve started or progressed over the last month.

Improving Our Quarterly Sprints and Goals #

This month our team kicked off the second round of our new quarterly sprints and goals process. This is an attempt to focus our team around a few goals and sub-teams dedicated to them throughout each quarter. Our hope is that this allows us to make strategic “pushes” in directions that feel important for 2i2c’s operations and the communities we serve.

This quarter, we incorporated a lot of learning that took place after our first iteration in Q3 2023. We are hoping to sharpen up the timing of events throughout the process, including communicating internally and externally about our status (thus, this blog post series).

In the next quarter, we aim to build off of this work, and to identify where our future new hires of Delivery Manager / Interim Chief of Staff and our Product Lead will fit in.

Three New Jobs Posted #

In October, we began a short hiring push to address many of the organizational challenges noted in our organizational structure and strategy audit. We aim to have each of these positions filled by the end of the year and begin incorporating new team members into our organization. Here’s a list of the three job postings:

Product Strategy Work #

We kicked off a collaboration with Richard Pope to provide us with some short-term product strategy work. In Q3, our team took stock of the many different kinds of services and technology that we deploy, aiming to refine this into a long-term product model.

Richard will join us for several months to take these outputs and help us craft them into a model for where 2i2c is delivering value that we can build upon for the coming years. We’ll provide more updates for the community as this work continues.

Continued Onboarding of Communities in Our Catalyst Project Collaboration #

We continued onboarding communities onto infrastructure managed by 2i2c as part of a CZI-funded project to serve communities in Latin America and Africa. The grant team began its operations last April and spent the first several months creating an onboarding pipeline and rubric for identifying and connecting with communities. As of October, we’ve onboarded our first few communities - there is still a lot of content, training material, and documentation to create, and we will begin iterating on this in collaboration with our early-adopted communities in Q4 2023 and Q1 2024.

Partnerships and Impact #

This section describes notable new partnerships and developments with communities in our interactive computing hub service.

We began several new partnerships with communities in the research space this month.

The NASA Visualization, Exploration, and Data Analysis (VEDA) project is an open-source science cyberinfrastructure for data processing, visualization, exploration, and geographic information systems (GIS) capabilities.

The US Greenhouse Gas Center provides a cloud-based system for exploring and analyzing U.S. government and other curated greenhouse gas datasets.

In addition, we’ve celebrated considerable growth in one of our partner communities: CryoCloud. This community focuses its work around studying the Cryosphere using satellite imagery data. They’ve grown their community hub to roughly 300 users in a year and have piloted several novel interfaces for data analysis in a JupyterHub context, such as using QGIS via a Linux desktop browser.

2i2c team members also gave several talks in October, and you can find links to each of these below:

Chris Holdgraf on 2i2c’s overall purpose and operations thus far, delivered at the Policy Simulation Library demo day.
Jim Colliander on the impact 2i2c wishes to have and what it means for open science, delivered to the ESIP community.

What We’ve Learned and What Stood Out #

These are things that we’ve learned in the past month that we’re using to improve our service moving forward.

Communities like to define and select their own environments. We began rolling out new functionality to allow communities to define their user environments on-the-fly. This brings mybinder.org-style environment creation into a JupyterHub, and gives both more control and more visibility into the environments that communities use. We noticed that this also begins to blur the line between different workflows and types of communities that can be served on a single hub. It means that a hub is no longer strictly tied to a small number of user environments created for it. We’ll explore the implications of this in the coming months.

We need to improve our billing and invoicing infrastructure. This month we began going through another backlog of cloud infrastructure invoices to send out to communities. For many communities, 2i2c pays a cloud bill on their behalf and passes through costs directly to them. This allows them to know exactly what they’re paying for. However, it is also starting to generate a lot of work! We aim to explore ways to automate and standardize this process in the future.

Digital Public Goods is a concept that resonates with many community leaders. Jim and Chris both gave talks that leaned heavily into the notion of open tools and services as “Digital Public Goods”. As public goods, it is important that they remain “driven by the public” and that they have sustainable and scalable models to grow the value that these goods provide. We’ve noted that this topic resonates heavily with people across the research and education spectrum, and we’re considering how to lean into these values more heavily in 2i2c.

Open organizational report: Strengths and challenges for 2i2c's team

Wed, 08 Nov 2023 00:00:00 +0000

Over the last several months 2i2c has been working with an organizational consulting group called Difference Digital to help us identify the major opportunities and challenges in our organizational structure and strategy. The result of this work is a report that describes in detail the major strengths, weaknesses, challenges, and opportunities that 2i2c faces. It also recommends major actions to take as an organization.

2i2c values organizational transparency and a willingness to be honest about where you’re struggling. Stress is a natural part of both individuals and organizations, and should be embraced with the goal of learning and improving. Moreover, as a young organization 2i2c has benefitted heavily from the documents and learning that other organizations have made publicly available. We wouldn’t be where we are without building on the backs of others who are willing to share what they’ve learned.

So, we are making this organizational audit and report public for anyone to see. We hope that it provides transparency into 2i2c’s current status, and that it serves as a useful resource for other non-profits that are growing and facing similar challenges.

Here’s a link to the report on Zenodo.

For more background on this report, check out the short description below.

In April of this year we had our first in-person team meeting. We were excited and grateful to have the opportunity to speak to each other face-to-face. We also learned that many people on our team were stressed out! Our service had grown slow-but-steadily over the previous year, and we were feeling the tension that comes with growing your partnerships without significantly changing your team’s capacity or structure.

So, we decided to work with an organizational consulting group called makeadifference.digital to help us identify where we need to make improvements. They spent several weeks having one-on-one conversations with each member of the team, as well as doing a broader organizational analysis and comparison to other technical organizations at a similar stage of their lifecycle.

The result of this work was a report that outlines the major opportunities and strengths, as well as challenges and gaps in capacity, that our team currently faces. It makes a number of recommendations for how we should shift our practices, and importantly it also notes that the only way to gain that capacity is by hiring for new people and skills.

Here’s a link to the report on Zenodo.

The immediate result of this work is that we currently have three jobs posted:

A Product Lead role to help us build a “product function” within 2i2c’s team.
A Delivery Manager / Chief of Staff role to oversee and manage 2i2c’s system of work.
A cloud and operations engineering role to grow our cloud engineering team’s capacity in serving communities.

Our next steps are to fill these positions, and to then begin the work of implementing many of the recommendations that are contained in the report. We’re confident that this is a natural part of being a small and growing organization, and we are grateful for the expertise of makeadifference.digital in guiding us through this work.

Note: if you’d like to work with makeadifference.digital, you can reach out to them at hello@makeadifference.digital.

Acknowledgments #

Support for organizational and strategic work like this is provided by a grant from CZI.

Community update Q3 2023: Service growth and growing pains

Tue, 31 Oct 2023 00:00:00 +0000

It’s been two quarters since our last major update - this isn’t quite as frequent as we’re hoping to post updates from our team, but we’re making adjustments to have more regular communication for reasons that will hopefully be a bit clearer below! In that time, we’ve been hard at work serving and growing our interactive computing service, as well as doing some introspection as a team and identifying major next steps moving forward. More on that in the following sections, but first a short service update.

How has our service evolved over the past few months? #

New partnerships and service growth #

Our service has grown several new partner communities over the last two quarters. A few notable communities are the NASA VEDA project, a team at the Smithsonian, a team at NCAR, and the U.S. Greenhouse Gases Center. We are running about 71 hubs across 24 clusters, with about 4000 weekly users ( more usage stats here).

We also began operations on a major collaboration to serve communities in Latin America and Africa, called the Catalyst Project. This team met together for the first time in April, and we’ve been laying a foundation for service growth in the first several months. We are just onboarding our first communities and hope to grow that service in the coming year.

Finally, we’ve been fortunate to receive some grants around creating content and designing workflows that utilize cloud infrastructure. For example, a NASA TOPS grant and an upcoming collaboration with Project Pythia around geospatial workflows.

Financial picture #

At this point, we’re recovering roughly 40% of our operating costs through recurring fees of our managed hub service (making up the remainder in development contracts and grants), and we’ve currently got around 2 years of runway.

However, both of these will be lowered soon because we are about to hire for several more positions. Improvements to our product model will allow us to estimate and recover our service costs more effectively, but we also intend to raise some funds next year to support our efforts in making our service more robust, sustainable, and valuable to communities.

It’s always difficult to strike a proper balance of team (and cost) growth against the financial buffer needed to assure your partners you’ll stick around, but we’re confident that the new hires described below will serve critical needs for our team and mission.

For more context on why and how we’re trying to make up that capacity, read on…

How service growth can lead to team stress #

Over the past year we’ve had a slow-but-steady stream of new communities interested in working with 2i2c for managing cloud infrastructure for interactive computing. We’ve taken a “let’s make it work somehow” approach to all of our community partnerships thus far, with the idea that we must use these partnerships to learn what communities want and identify common patterns.

This is exciting, and we’re fortunate to see the interest and growth in our service. It suggests to us that something about our model is fundamentally right. Communities really love the Right to Replicate, and our participatory service model based around upstream contributions, transparency, and shared responsibility is attractive to many in research and education.

However, each new community is also a new set of stresses on the technical and social infrastructure of our team. Without the capacity to manage the demands of the service, you run the risk of over-extending – and in a worst case scenario, burning out – your team.

This became clear in our first in-person team meeting last May. At that meeting, we realized that many on the team were spending too much of their time “reacting” to demands from the service. We also learned that the scope and complexity of our various workstreams had gotten to a size where our informal team structures of work prioritization were no longer adequate.

Growing the complexity of our team to match the complexity of our service #

So, for the past several months we’ve been working on a plan to evolve 2i2c’s team structure in order to more effectively manage the complexity of our service, and balance long- and short-term thinking.

This began with an organizational audit carried out by makeadifference.digital, a consulting group that focuses on tech-for-good and non-profit products and services. They conducted interviews with everybody on the team, and concluded that we have a few key functions missing that were creating or compounding the stresses people felt.

We hope to make some of their key findings public soon (Update: this is now available at this blog post about the organizational report), and in the meantime here is an overview of some highlights:

We need a dedicated product functionality #

First, we realized that we have a number of new “signals” pushing our service in many different directions. Some are external - from communities we work with or from funders. Some are internal - from different team member visions of where work is needed. After growing our service and our team, there are now too many perspectives and voices to balance in an unstructured and purely organic way.

So we’re creating a dedicated “product function” within 2i2c. Its goal will be to serve as an integrator across the many stakeholders that are interested in 2i2c’s service, and synthesize a collective product strategy and system of prioritization to move the organization forward. They’ll help 2i2c grow a culture of iteration and decisions that are driven by the needs of our community partners and users.

We’re hiring a Product Lead to kick off this effort. If you think you’d be interested in this role, please consider applying!

We need a more structured system of work and execution #

Next, we realized that there are many different workstreams we must balance simultaneously. There are new communities to be onboarded, infrastructure bugs to be fixed, questions to be answered, and strategic priorities to improve the service. Expecting each team member to individually find the right balance between all of these is increasingly unrealistic. We need capacity to oversee this system of work and bring some structure to it.

So, we’re creating a Delivery Manager role to oversee the system of work that our engineering team uses for its planning and execution. In addition, this role will serve at an organization-wide capacity as an interim Chief of Staff. This will allow them to improve and standardize practices organization-wide, and coordinate major projects that span our team.

You can find a job posting for a Delivery Manager / interrim Chief of Staff role. If you think you’d be interested, please apply!

We need more capacity to support our team and its individuals #

In addition to moving forward our workstreams, we also have a unique challenge in team support and dynamics. 2i2c is a small team, it is also distributed across 10 time zones! This adds a lot of extra challenge in getting the team to communicate with one another, support one another as individuals, and learn and grow as a team. We believe that it’s critical to support one another, and to have systems in 2i2c that recognize these challenges in distributed collaboration to design around them.

So, we’re exploring how to create a “People Operations” function within 2i2c that can dedicate their time to supporting 2i2c’s team as a group as individuals. This might mean defining creative ways for team-building, creating mechanisms to surface where team members are struggling or need support, and growing best-practices in inclusive asynchronous team culture.

We’re in the process of designing this role, and hope to have it out in the coming months!

We need more engineering capacity #

Finally, we also realized we need more cloud engineering capacity on our team, particularly in a timezone that overlaps with the American continents. This will give our team a bit more breathing room to balance reactive and project-based work, and have more capacity to focus on personal growth and learning.

You can find our Open Source Infrastructure Engineer job post here if you’re interested.

What’s up next for our team? #

Finally, we’ve also adopted a new quarterly planning system to align our team around key goals that deserve focused attention in each quarter. This will let us be more intentional about considering our work in the context of 2i2c’s broader opportunities and challenges.

For the final quarter of 2023, we’re focusing on these key goals:

Begin to standardize and refine our product offering. In advance of our Product Lead role, we are doing a bit of work to more precisely describe and structure our current service offering. Our goal is to lay a foundation that our upcoming Product Lead can then use to learn about 2i2c’s product and make progress more quickly.
Pay down technical debt. Our engineering team is going to focus on streamlining their operations and reducing toil, in order to reduce the number of “unexpected distractions” that come with inefficient infrastructure and processes. It will also make the service more reliable and transparent for our communities.
Build foundational resources for hub champions to support their community. Our Partnerships team will lead an effort to build basic infrastructure and content that communities can use to create their own knowledge bases around hubs. This will be ongoing work so the goal here is to create a foundation to build upon in the future.

We hope that this has been an informative update from our team, and we appreciate everybody’s input and collaboration over the past few months. We think that the tensions described in this post are healthy - they’re reflective of an organization that is growing and reaching organizational milestones. We’re excited to tackle them and to continue growing our impact and partnerships.

Acknowledgments #

Support for organizational and strategic work like this is provided by a grant from CZI and fees from our Member Communities.

Community update Q1 2023: Growing our community partner network and our team

Fri, 28 Apr 2023 00:00:00 +0000

It’s a month after the end of Q1 2023, and we’d like to share a belated update about what we were up to in the first quarter of this year (we have good excuses for being late, including new tiny humans, I promise).

This quarter we grew our engineering team significantly, and started to refine our team processes and structures to accommodate this extra complexity. We expanded our managed cloud service with new community partners, and made a number of improvements to our technical infrastructure and organizational processes for managing this service.

Read on below to learn more about what we’ve been up to!

New community partnerships #

We added several new community partnerships to our managed hub service. We’ve deployed new hub infrastructure for each of the following groups:

Our University of Toronto Jupyter service now has a dedicated R hub.
We ran an event hub for the Drakkar Ocean project
We deployed a hub for the NASA Visuaslization Exploration and Data Analysis (VEDA) project.
We deployed a hub for the QuantifiedCarbon organization.
We ran an event hub for OceanHackWeek 2023.
We onboarded several new community colleges to our “JupyterHubs Education in Community Colleges” collaboration with CloudBank and UC Berkeley CDSS.

Service improvements #

Below are a few highlights for ways in which we improved our Managed Cloud Service for our partner communities.

We simplified our authentication workflow with CILogon #

Authentication services allow us to identify a user when they log onto a hub, which determines their ability to access hub resources. Previously we had used a combination of Auth0, CILogon, or GitHub for authentication.

However, over the past year we’ve been happy with our use of CILogon so far, especially because of its non-profit status and alignment with many research and education institutions that we work with. This quarter, we decided to streamline our authentication process by dropping the use of Auth0 and grow our partnership with CILogon.

💡 Learn more

We exposed user activity dashboards in JupyterHub so communities know how many people are using the service #

We tend to work with leaders of communities that utilize our service and infrastructure for many others in their network. For example, a teacher with a classroom of students, or a researcher with a global network of collaborators.

In these cases, it’s useful to track how many users are actively using a platform over various metrics of time. This can tell you whether your community finds a service useful, and whether this is growing or shrinking.

We use Grafana to automatically generate dashboards of activity for all of our community hubs and clusters. However, tracking daily, weekly, and monthly active users was not part of JupyterHub’s core functionality.

So, we decided to upstream this functionality into JupyterHub and expose it via the JupyterHub Grafana project. All 2i2c hubs now track daily, weekly, and monthly unique active users. And importantly, anybody else deploying the Zero to JupyterHub for Kubernetes can use this feature now too.

💡 Learn more

See Yuvi’s blog post about this feature.

We’ve added a support widget to our service documentation site. This will allow users to provide structured support requests directly to our team, allowing us to triage and respond more quickly.

Here’s our support button and widget in action:

Our fancy new support button!

We upgraded Kubernetes across all of our AWS clusters #

Kubernetes is at the core of our cloud infrastructure and scalability. We use either a shared or a dedicated Kubernetes cluster for each of our community partners, and it is the foundation upon which all of their Jupyter infrastructure rests.

One of the biggest challenges with managing an ongoing cloud service is keeping the underlying infrastructure upgraded. This brings in new stability and functionality, but also often involves manual steps and toil. This quarter, we upgraded each of our AWS JupyterHubs to Kubernetes 1.24 and will continue this effort with other providers in the coming quarters.

We streamlined our hub uptime checks to be more efficient #

The best kinds of failures are ones that your operations team recognizes and solves before any users run into the problem themselves. We use the Google Cloud Platform HTTP uptime checker to run regular uptime checks for each of the community hubs that we use. This allows us to get quick alerts if any of our community hubs is down for some reason.

We made several optimizations to this process so that we can more efficiently monitor hub uptime and trigger alerts to our engineering team if action is needed.

Organizational updates #

We made a number of broader improvements to our team processes and policies, and even got a shout-out from a few community partners!

We defined organizational principles around engaging in healthy ways with open source communities #

2i2c interacts with many open source communities throughout its work, due to the fact that the majority of the improvements we make are done via upstream contributions. We define several key open source communities that we interact with, and we needed a philosophy and set of practices to ensure that our relationship with them remains healthy and supportive.

💡 Learn more

Check out Principles and considerations for ethically accepting funding for open source for our first take on this, and our Open Source Strategy page for our current team policies around this.

We responded to a NASA RFI on “Scientific Data and Computing Architecture to Support Open Science” #

NASA recently put out a Request for Information around open science and open source infrastructure: NASA RFI NNH23ZDA005L: Scientific Data and Computing Architecture to Support Open Science. We responded with some of our ideas around how to build more open, inclusive, collaborative scientific communities with cloud infrastructure and open source tools.

💡 Learn more

Check out our submission, “Building the open source stack”.

We added two new team members and a new team role #

Our Product and Services Team continues to grow! We added a new team member to improve and manage our cloud infrastructure service: Erik Sundell. As our team has grown, we have also had to manage more complexity in communicating and coordinating across a globally-distributed team of engineers. To help us manage this system, we’ve created a new Engineering Manager Role and promoted Damian Avila as our team’s first engineering manager.

OpenScapes wrote up a blog post about our cloud service collaboration #

We got a nice shout-out from Luis Lopez at the OpenScapes project, describing the benefits of the cloud service that we run for their communities. Check out his blog post: The why, what, and how of our NASA Openscapes cloud infrastructure: 2i2c JupyterHub and corn environment.

Onward to Q2 #

Many thanks to the 2i2c team, our partner communities, our funders, and the many others that have provided us support and guidance. We hope that this update provides a helpful idea of our priorities and major efforts, and we look forward to giving you a new update in Q2!

Acknowledgments #

Support for organizational and strategic work like this is provided by a grant from CZI and fees from our Member Communities.

2022 in review: growing our partner communities and expanding our operations

Wed, 04 Jan 2023 00:00:00 +0000

2022 was a busy year for 2i2c - we not only grew our operations as well as our organization, but also grew our understanding of our mission and where we can have impact. This is a brief reflection on this experience, and an attempt to identify our opportunities for impact and growth in 2023.

Our major goals in 2022 #

We wrapped up 2021 with two major new changes. We had just finished moving fiscal sponsors and had just finished a prototype of our alpha service offerings.

Our 2x2 matrix of service offerings and prices created at the end of 2021. See the documentation for more details.

Our biggest challenge in 2022 was to identify the bottlenecks in this service model, and to begin building the infrastructure to operate and scale it. This included team infrastructure, technical infrastructure, and administrative infrastructure.

Let’s see what we did to accomplish this goal.

Highlights from 2022 #

In 2022, we thoughtfully grew the number of communities we worked with, and used this to make iterative improvements in our model. As a result, we learned some important things and made significant improvements to our service model and infrastructure. Here are a few highlights.

We grew the number of our partner communities #

As noted above, we needed to grow the number and diversity of communities we worked with to understand where our model needed to change. At the end of 2022, we now have 43 community partner hubs across 17 clusters (and at least one on AWS, Azure, and Google Cloud). This amounts to roughly ~2,500 active users each week. We also ran more dedicated infrastructure for more than 11 workshops and events.

We grew our revenue from community partnerships #

One of our goals is to reach self-sustainability without requiring grant funding for most of the communities we serve. In 2022 we built administrative infrastructure to more efficiently recover monthly costs, and were able to bring in funding for our team from community partnerships. Here’s a plot of our monthly non-grant revenue over the last several months:

Our monthly non-grant revenue over the last several months. June is much larger because we filled a backlog of invoices from previous months that weren’t billed yet.

We got grant funding to serve communities in Latin America and Africa #

We also learned that some partnerships may require subsidization from a third party, such as historically marginalized communities and those without dedicated resources. To explore sustainable ways to serve these communities, we applied for and received a new grant to serve communities in Latin America and Africa! Here’s the blog post announcing this grant and our open grant narrative from the proposal.

We improved our continuous integration and deployment system #

Our ability to sustainably grow our service requires being able to technically serve many communities from a relatively small team. We centralized and standardized configuration and operations of many community hubs in one transparent space for all of our partner communities. This allowed us to more easily grow the number of communities we served from one repository. You can read a write-up about these improvements in this blog post.

We defined a Shared Responsibility Model #

Our goal is to frame each community hub as a partnership with a clear breakdown of responsibility to give communities more agency over the infrastructure and service. The Shared Responsibility Model provides a framework for assigning responsibility for various tasks with our partner communities. See our Shared Responsibility Model docs here.

We defined a formal Incident Response process #

Cloud infrastructure inevitably degrades over time, and running ongoing services is largely about quickly responding to issues and resolving them quickly. To do so, we need clear processes to follow in order to quickly identify and respond to major incidents in the infrastructure. Our Incident Response process defines formal team roles and alerting mechanisms that are served by PagerDuty, following best-practices in industry. This will make our service more reliable and make our processes more transparent for our partner communities. Here’s our current incident response process.

We expanded our service offerings to include community and workflow guidance #

We recognized that many communities need more than just infrastructure running in the cloud - they will also benefit from usecase and community guidance. We’re exploring a new range of roles that we could fill, starting with hiring a new team member to help us lead these efforts. Here’s a blog post about the Product and Community Lead.

We began a collaboration with GESIS to develop environment building in JupyterHub #

This marks our first efforts into development-focused work as opposed to operating cloud infrastructure. We will use this experience to learn how to pair focused development with cloud operations (more on that below). It will also make it more likely that we can implement often-requested improvements to the JupyterHub / BinderHub ecosystems. Here’s a blog post about this collaboration.

We helped maintain several upstream open source projects #

We made a number of contributions to key open source communities as part of our organizational mission. These spanned technical improvements as well as organizational and community efforts. One highlight is that several team members have participated in JupyterHub’s latest round of Outreachy interns.
See the JupyterHub community strategy page for more information.

As a brief summary, here is a plot of the issues in key open source repositories that were closed in 2022 that were authored by a 2i2c team member.

Upstream issues opened by a team member that were closed in 2022.

We refined our organizational strategy #

Finally, the experience from this year gave us a lot to think about regarding our role and potential for impact in the research and education ecosystem and the open source community. We updated our strategy in order to focus on a more holistic and collaborative approach to the work with our partner communities. We’ll continue to refine this strategy moving forward.

Things to accomplish in 2023 #

We’re excited about all of the progress we made in 2022, and to continue that progress in 2023. Here are a few areas where we wish to focus our efforts as we begin the new year to keep this momentum going.

Sharpen our Shared Responsibility Model #

Our Shared Responsibility Model is in a kind of “alpha” phase right now. We have some of the high-level skeleton there, but there’s a lot of detail to fill in. Thus far, communities have really liked the idea but we need to make it clearer how we break-down specific jobs and how to decide when to give more or less responsibility to another community. We’d also like to define more intersection points with our partner communities at the level of strategic and service planning so that our communities have a say in our vision and strategic plan.

Grow our community support operations #

We’ve begun exploring how to support communities in their usage of cloud infrastructure, but still have a long way to go to understand what role we should play here. For example, how can we assist communities in cloud workflows without becoming domain experts ourselves? How can we guide communities in a sustainable and scalable way? How can we recover the costs of doing this work? We hope to use our experiences in our upcoming project to serve communities in Latin America and Africa to explore these questions.

Define a cost recovery model that balances sustainability and accessibility #

Our current cost recovery model is intentionally very simple - we charge a fixed monthly cost for human time, and pass-through cloud costs directly. We charge a bit more for more complex deployments and use-cases.

However, we’ve learned that this is both too expensive for simple deployments, and too cheap for really complex deployments. It’s also inaccessible to organizations that do not have the resources to pay, which correlates heavily with historically marginalized communities. We’ll need to refine this model to be both scalable and sustainable, but also accessible to the variety of communities we want to serve.

Incorporate a dedicated software development practice that aligns with open values #

In 2022 we learned that our position in running infrastructure in the cloud gives us visibility into the ways that researchers and educators want open source tools to be improved. In some cases our partner organizations are also willing to contribute resources to help make this happen, or we can identify third parties to fund development work.

Thus far we have focused our efforts on deploying and managing cloud infrastructure. We do make improvements to software as a part of this, but there’s a big difference between running a cloud service and making significant enhancements to software.

We’d like to use our position to help funnel more resources into open source development, but there are a few tricky things to figure out. For example: How can we accept funding to do open source work in a way that doesn’t effectively make us sole decision-makers or gatekeepers for doing development? How can we incorporate software development team practices into a team that has thus-far focused on operating cloud infrastructure?

Thanks for reading this update about our work in 2022. We’re excited about what we’ve accomplished thus far, as well how we hope to expand our impact in the near future.

Acknowledgments #

Support for organizational and strategic work like this is provided by a grant from CZI and fees from our Member Communities.

Celebrating our progress in Q3 2022

Sun, 16 Oct 2022 00:00:00 +0000

Quarter 3 of 2022 has wrapped up, and the 2i2c team has been busy making improvements across our infrastructure, organization, and operations. This is a quick post to celebrate the work we’ve done over the past three months, and to briefly share what we’re working on next.

Below we’ll provide a brief update about major developments this quarter, broken down by functional areas of 2i2c.

These are the main highlights from this quarter - if you’d like to check out more of the work that we’ve done, see:

All the PRs we’ve merged in Q3

All closed issues in Q3

Community impact #

These are a few ways in which we’ve collaborated with communities and demonstrated impact over the last few months.

New JupyterHubs for communities. We’ve deployed JupyterHubs for several new partner communities. Here’s a quick list:

PaleoCube and PaleoHack Hubs infrastructure#1418
NeuroHackademy 2022 infrastructure#1505
Alabama Water Institute CIROH hub infrastructure#1444
OceanHackWeek 2022 infrastructure#1515
COESSING Pangeo-Style Hub infrastructure#1516
Temple University Education Hub infrastructure#1648
Callysto Hub infrastructure#1439
London Interdisciplinary School infrastructure#1485

We also ran hubs for several community events:

NeuroHackademy: infrastructure#1300
OceanHackWeek 2022 infrastructure#1576
COESSING workshop: infrastructure#1516
Eddy Symposium: infrastructure#467
Allen Institute Summer Workshop on the Dynamic Brain infrastructure#1621

Please see our service documentation for more details.

Organization wide updates #

These are large-scale organizational and strategic efforts that impact all of 2i2c.

We applied for a CZI Grant: In partership with The Carpentries, CSCCE, MetaDocencia, Invest in Open Infrastructure, and OpenLifeScience, we applied for a CZI grant to provide cloud infrastructure services to global communities.

We grew our team: We’ve hired two new team members to lead new major efforts with 2i2c. James Munroe will lead efforts around community guidance and product design, and Jim Colliander will lead efforts around partnerships and sustainability. We also updated our Hiring and Candidate Search documentation in the process.

We’re refining our strategy: We’ve begun a process of revisiting and refining our strategy after a year of major operations, see our strategic update blog post for more information.

We completed the CSCCE community management training. Two of our team members (James and Sarah) both completed a several-week community management course that was offered in partnership with CZI.

Our team member Sarah began a part-time role as the JupyterHub Community Strategic Lead. Sarah will be leading community strategy efforts within JupyterHub for the next two years thanks to a grant to the JupyterHub team from CZI. Check out this issue to follow our progress.

Service improvements #

We made a number of improvements to our cloud infrastructure and the processes around our service. Here’s a brief breakdown:

We expanded our shared clusters to new cloud providers and regions. We now have shared clusters already deployed on Google Cloud Platform on us-central1-b and europe-west2.

We defined an incident commander process. This will allow us to coordinate and respond to major outages in our cloud infrastructure more efficiently. See our incident response documentation for more information.

We improved our cloud usage monitoring infrastructure. We’ve deployed a centralized Grafana Dashboard that aggregates cloud usage across all of our partner communities, and allows us to keep track of any unexpected behavior or outages across them all.

Where we’re focusing next #

In the final quarter of this year, we’ve decided to focus our efforts on growing capacity across all of the aspects of our team. Now that we have brought on several more partner communities into our Managed JupyterHub Service, it has shown us where we have bottlenecks in our technology, process, and structure. In 2023 we hope to significantly grow the number of communities we work with, and so we must grow our capacity to be able to take on these new partnerships.

We aim to accomplish this in a few ways:

Make technical improvements to our cloud infrastructure that reduces the amount of human labor associated with regular actions. This will make our cloud infrastructure more scalable and reliable.
Improve our invoicing and partnership leads pipeline so that we can reduce the amount of administrative toil for ourselves and for the communities we work with in billing and cloud cost pass-through.
Refine our organizational strategy and structure so that we are better-able to agree on our most important objectives, and to execute on them efficiently as a distributed team.
Hire people! While improving our efficiency will certainly grow our capacity, we also hope to grow our capacity the old fashioned way: by hiring more team members. We’re identifying the biggest needs on our team now and will hope to have postings soon. Stay tuned!

Thanks #

If you are interested in partnering with 2i2c to have your own managed JupyterHub, please contact us at partnerships@2i2c.org. We have a shared cluster on Google Cloud, with plans to deploy one on AWS soon, and dedicated clusters can be run on any major cloud provider. Please see

our service documentation for more details.

One year later: an update of 2i2c's mission, strategy, and impact

Sun, 09 Oct 2022 00:00:00 +0000

This post is an exploration of 2i2c’s current strategy and direction after a year of major operations. It is a brainstorm from the Executive Director, shared as a blog post to invite feedback and provide transparency into our current thinking. Its goal is to explore the context of 2i2c’s stakeholders and their needs, and identify an opportunity and plan for having a positive impact with these communities. It is not a concrete proposal but a snapshot of thinking in time meant to trigger reflection. Over time we will incorporate some of these ideas into our Team Compass.

When we founded 2i2c, we largely did so from the “bottom up” - we identified several patterns around hosted infrastructure that were useful at UC Berkeley, Pangeo, and similar communities, and we wished to make them more generalized, accessible, and scalable.

We defined our mission as the following:

Our mission is to make research and education more impactful, accessible, and delightful by developing, operating, and supporting infrastructure for interactive computing.

And a description about our immediate activities to make things a bit more concrete:

2i2c designs, develops, and operates JupyterHubs in the cloud for communities of practice in research & education. It builds and supports open source infrastructure that serves these communities.

Around a year ago we began our pilot JupyterHubs project to learn more about our biggest challenges and opportunities in making interactive computing more accessible and useful for research and education. While both of these statements are still accurate, over the past year we’ve also learned more about the value that 2i2c provides. This post is an exploration of how these statements and our strategy may evolve in the near future.

What did we miss with our original strategy? #

In short: it is too-focused on actions rather than impact.

While running JupyterHubs is a key part of what 2i2c does, it is a means to an end rather than our end-goal. Infrastructure is only useful if it changes workflows in a way that aligns with the goals and values that we wish to achieve. We’ve historically defined these in a few scattered places. For example, here are the values listed on our website:

2i2c values fairness and justice as requirements for successful communities.

2i2c values learning and discovery for all people.

2i2c values collaborating and connecting to foster environments for learning and discovery.

However, it is difficult to tie our operations directly to values and goals without making them concrete, and without defining a plan that ties them to our work.

Through our JupyterHubs pilot, we’ve learned how our actions-focused approach was missing some important aspects of these broader goals. For example, we came to understand that a big part of 2i2c’s value isn’t just providing a JupyterHub, it is de-risking cloud native workflows for communities that are inherently skeptical of what cloud infrastructure offers. Doing this entails many things: teaching, making decisions on behalf of others, supporting and answering questions, building trust, and yes, managing infrastructure.

Refining our strategy #

With this in mind, we’d like to make our strategy more clearly-defined and tied to our operational choices. Here are a few ways we’d like to do this:

Define our organization’s values and vision for the impact we wish to have.
Define the key stakeholder communities that we wish to serve, the context of tools and services that are relevant to these stakeholders, and the assumptions we’re making.
Define the problems these stakeholders have, the ways in which their current workflow could be improved, and the opportunity to help them.
Describe our strategy to positively impact these stakeholders with our work.
Define a collection of goals and objectives to carry out this strategy in the near-term.

The rest of this post will take a crack at answering a few of these questions. It is intentionally messy, and meant both as a public snapshot of my thinking at this moment, and fodder for discussion and more specific proposals in the future.

Context: Our key stakeholders and the impact we wish to have #

2i2c’s key stakeholders are communities of practice that are dedicated to creating and sharing public knowledge. These are primarily made up of researchers and educators in the global community.

For these stakeholders, we wish to catalyze and support a transformation in their data workflows that allows them to be more collaborative, inclusive, efficient, and powerful in the impact they wish to have.

Assumptions we make about our stakeholders #

There are a few unique things about these communities that are important for us:

Researchers and educators see their job as creating and sharing knowledge with a heterogeneous and global community. They can’t make many assumptions about the organizational context or resources of this community, or their work will become inaccessible to others.
They work at vertically-oriented institutions (e.g., a university with an in-house IT department), but their communities are organized horizontally across institutions (a researcher may identify more with colleagues in their field than with their co-workers at the university).
As a result, these communities value workflows that are maximally accessible, portable, modular, simple, and long-lasting. This allows them to define a shared set of practices across their institutional boundaries without requiring a lot of vertical decision-making.

Historically, this means that scientists must choose between workflows that are underpowered but accessible (e.g., collapsed into a PDF describing the work), or powerful but inaccessible to many (e.g., they rely on extensive training or resources only accessible to some institutions).

Open source tools have boosted standardization and collaboration across communities #

Over the past ten years, the proliferation of open source tools has vastly expanded our potential to create public knowledge in a more accessible, modular, inclusive, and productive way.

For example, core numerical computing infrastructure like NumPy provides a foundation for more domain specific numerical libraries to build on. Tools like Pandas and XArray allow for more sophisticated data structures that match the complex and heterogeneous data in the research community.

This has led to many domain-specific communities of practice building upon the same APIs and infrastructure to solve their specific needs. By largely sharing the same stack, we have reduced duplication of effort and make it easier to speak the same language across data-rich fields.

…but a local workflow with open source tools is inaccessible or underpowered for many people #

However, this has also led to a few key problems:

Problem 1: The ecosystem of open source tools is huge and complex, and hard to tailor for specific domain workflows. It is hard to decide which tools to use for a workflow, and we do not have efficient community processes for creating and standardizing workflows.

Problem 2: For many users, it is still too difficult to set up a local environment. Installing the right libraries and their dependencies is complex and domain-specific. This leads to wasted time and confusion, especially for newcomers or learners in an educational context.

Finally, there is another workflow-related problem that arises even when you do set up a proper environment:

Problem 3: Local workflows are not scalable enough for many communities. Working with 10GB of data can be challenging in a local context, working with 10TB of data quickly becomes impossible. This leads to communities that “share data” by making lots of local copies that are distributed to individual users. This harms data provenance and makes our work less reproducible and accessible. It limits the scope of questions we can ask with data and increases the friction for sharing our work with others. As our workflows require more computation and data, a “local, fixed hardware” approach will become more and more underpowered.

Managed cloud services can solve the accessibility problem for many workflows and tools #

One way to resolve these problems is to rely on managed infrastructure services to provide the environments needed for data analysis. A systems administrator can set up computing and data resources, and then expose these resources to many users as a service. This makes these complex and powerful environments more accessible, standardizes the workflows of users to make it easier for them to collaborate, and potentially serves as a gateway to more complex data and computing infrastructure.

Historically, this work was done in-house by internal teams at research institutions. This led to many unique infrastructure setups with slight differences across institutions. It also led to data and workflows being institution-centric, even though many communities are organized across institutions.

Cloud infrastructure can solve this problem. It provides a layer of infrastructure that can be used to standardize access and worfklows across communities of practice. It can be more accessible than on-prem infrastructure, as the cloud is already globally distributed in a way that local hardware can never be. Moreover, cloud infrastructure is also much more scalable, and can be used to access more sophisticated data and computing resources in a fluid manner.

But the cloud services landscape is not designed for research and education communities #

Over the last 10 years, a huge ecosystem of cloud-native tools and services have grown to solve this problem in industry. These are often called the Modern Data Stack. However, modern cloud services are largely unused by research and education. Why is this?

We believe it is because most cloud services do not meet many needs of research and educational communities:

They tend to be offered by a single vendor, which results in a fear of lock-in or vendor-specific workflows that result in high-costs.
They require users to learn non-standard workflows that are attached to the cloud service.
They often require extensive knowledge of data and cloud resources to configure properly. They may require an internal dev-ops team to integrate cloud services and expose them for internal teams.
They are often still institution-centric because cloud services are geared towards enterprise contracts within vertically-integrated organizations. Many research organizations are more like collections of little fiefdoms rather than a single kingdom.

In short, many cloud services ask researchers and educators to give up too much agency and choice, and make them reliant on organizations they do not inherently trust. They are perceived as a risky alternative to the “we have control and have visibility into everything” nature of on-prem hardware.

Examples of a potential model for community-centric cloud services #

Over the past several years we have seen several communities strike a balance that made cloud infrastructure more accessible while still retaining agency for the communities of practice that used and managed the infrastructure. For example:

The Pangeo Project is a cloud-native geospatial analytics platform that builds on the Dask, Zarr, and XArray stack to enable distributed and collaborative access to large-scale geospatial datasets.
The UC Berkeley DataHub is a cloud-native educational community that uses a managed service to provide interactive learning environments to thousands of students via the cloud.
The Syzygy service is a federated network of educational hubs that run entirely on open source infrastructure in the cloud, managed for universities in Canada.

Each of these communities roughly followed the same model:

Integrate a collection of open source tools and cloud services into a standardized cloud environment for a community.
Customize the environment and cloud resources via open source tools that provide remote interactive computing sessions. This is usually done with JupyterHub.
Contribute to upstream tools and redeploy improvements to a community’s shared infrastucture in order to enhance the service.

Moreover, they use nearly identical technical stacks: some combination of Jupyter infrastructure to manage cloud infrastructure and provide interactive computing environments, a Scientific Python or R-based stack for the libraries users access, and a collection of online services to facilitate collaboration and workflows.

These services required a combination of dev-ops and cloud engineering skills to integrate the tools and manage the infrastructure, as well as community guidance and teaching to standardize and share workflows across a community.

So why couldn’t this model just be repeated across many other communities as well?

However, running your own cloud infrastructure is a lot of work #

These initiatives demonstrate the potential of building community-centric cloud services on top of open source tools, but they were also unscalable to other communities. Observing these success stories also reveals a few more problems that we must resolve in order to make these services more generalizable and accessible to new communities:

Problem 4: Running your own cloud requires skills and resources that most organizations do not have. In order to make the most of the cloud and to provide it as a resource reliable-enough for important workflows, it is important to have a knowledgeable team with best-practices in distributed cloud management. However, hiring engineers in this space is very difficult, and the team structure and culture needed for these services is often at-odds with the way universities are structured.

Problem 5: Contributing to open source communities requires dedicated time and skills. When building services on top of open source tools, you will invariably need to improve the tools in order to fix bugs or to enhance your service. Doing this is very different from a contract with a traditional vendor, and requires building skills in how to communicate and interact with open source communities. For key open source infrastructure, it also requires an ongoing investment of time into upstream community work that is hard for many institutions to justify.

Problem 6: When institutions run their own clouds, they tend to re-invent institution-centric anti-patterns. For example, an institution might expose a cloud-based service but make it available only to individuals with an institutional account. They might staff it with an under-resourced team (or just a single person). They might make small decisions that differ from institution to institution. All of this limits the effectiveness of the cloud for facilitating collaboration and sharing within communities of practice.

Finally, while vendor-specific cloud services often come with a lot of training material, self-managed cloud services require institutions to build their own practices and guidance for how to use the infrastructure. This results in another major problem:

Problem 7: Communities need assistance in understanding what is possible with cloud-native workflows, and adapting these workflows to their domains. There is enormous value in integrating tools together and exposing them as a highly-accessible managed service via the cloud. However, most communities of practice are still new to cloud-native workflows, and they need considerable guidance to begin sharing their practices within a community.

These are several major problems that we must overcome in order to make cloud infrastructure more accessible and useful to research and education communities while living up to our values of community agency and open workflows.

Our opportunity for impact #

To summarize, we believe that there is the following opportunity for impact:

Make communities in research and education do work that is more collaborative, reproducible, and impactful…
In a way that gives them agency and flexibility to do their work as they wish…
By integrating open tools and standards, managing them as a service with scalable cloud infrastructure, and guiding communities in deciding how to use these tools for their domain-specific problems…

We believe that the best way to do this is via a mission-aligned non-profit service that is based on values of trust, shared responsibility, and transparency. This will allow us to treat our relationships with communities as collaborators, not customers and share responsibility and agency with them. In this way we can act as a vehicle for sharing resources and ideas across communities via our infrastructure and services.

We can make such a service both sustainable and scalable because we are delivering direct value to communities with resources. There are many potential business models that are already in existence (for example, Software as a Service or shared research infrastructure services like supercomputing centers). By generating resources in exchange for these services, we can centralize cloud infrastructure services for many communities into a single, scalable team. By committing ourselves to an upstream-first service model, we can drive resources towards open source communities and make these tools more useful for our key stakeholders.

A model like this naturally sets up a two-directional virtuous cycle where open source communities and public knowledge communities derive value from one another’s work.

flowchart TD RE[Communities in
Research and Education] OS[Open Source Communities] OS -- Builds open tools and
standards used by --> 2i2c 2i2c -- Integrates tools into
managed services for --> RE 2i2c -- Integrates feedback and
upstreams improvements to --> OS RE -- Provides feedback
and resources for --> 2i2c

How should 2i2c try to achieve this impact #

With this in mind, we hope to focus our organizational efforts around building services that focus on the following actions.

Integrate open source tools and services into a cloud-native service that is optimized for scientific workflows and communities. The end-product should not feel like “a 2i2c product” but a service anybody with enough skills could build themselves.
Manage cloud infrastructure with an “upstream first” mentality to provide a high-quality and accessible service for global communities. This will allow communities to focus on their work instead of the infrastructure.
Support the communities that use this infrastructure so that questions and problems are dealt with quickly. This will de-risk the service (and the cloud more generally) with the knowledge that help is available.
Guide communities in using these tools to do their work more effectively and collaboratively. This will speed up the learning and sharing process so that communities are able to make the most of the cloud.
Collaborate with these communities and embody the values we want to see in the research community to build trust and lead others. This will build trust and strengthen relationships between our communities.

Over time, we may add new verbs (such as develop open source software and lead open source communities), but the ones above define our initial organizational focus.

Our most important actions to take #

Accomplishing this requires blending aspects of cloud infrastructure management, domain and workflow expertise, and collaborative service design. As we move forward, these are the biggest challenges to overcome in developing this service further:

Build a top-notch global Site Reliability Engineering Team. First and foremost, the infrastructure that we manage must be reliable and scalable. If you ask communities to do their work in the cloud, they must be confident that it will be stable and highly available. If you wish to serve a global community, then you must build systems management capacity on a global level. This means investing heavily in our asynchronous coordination and communication processes, and building team skills in the modern cloud infrastructure stack and dev-ops practices.
Build centralized cloud deployment infrastructure for community-first services. We need the ability to centrally manage infrastructure that is delivered in a community-specific manner to many different communities of practice. This will allow us to achieve economies of scale in managing and running the infrastructure, while allowing communities to have agency over their tools to customize them for their needs. To do this we’ll need to use open source tools and cloud services that are designed for flexibility and modularity, create out-of-the-box infrastructure that gets communities 80% of the way there, and gives them the power to define the last 20% in a way that is sustainable for both them and our team.
Define two or three scientific workflows to focus on, and integrate the right tools and services into a distribution for each. We must identify a few scientific workflows to focus on, as well as the infrastructure stack that will be most-useful to the communities that require these workflows. We must integrate this infrastructure into a distribution that any community could pick up, but that has enough flexibility for each to solve their “last-mile problem” with customizations.
Define a support and guidance model for community-first services. Our support and guidance similarly needs to be generic enough to be scalable to many kinds of communities, but with the ability for communities to make their own customizations and have agency over their training and content. In this way we can be a centralized resource that communities can use to grow their own practice of cloud-native workflows.
Identify a collaborative service delivery model that doesn’t create too much overhead. We believe that community-centric infrastructure should be transparent and participatory, and provide agency to communities in shaping their service. On the other hand, we wish to serve many different communities, and thus we must minimize the amount of labor associated with serving any particular community. As a result we must build a collaborative model of roles and expectations around our services that balances participation from community representatives against our ability to act quickly to ensure the best service possible.
Identify a cost-recovery model that is globally accessible and scalable. We wish to serve a large and global collection of communities with our services, and so we must identify a way to recover our costs such that we are not restricted to serving only the wealthiest communities in the world, and such that “cost vs. resources generated” gives us the flexibility to grow to serve a large global community.
Create an upstream-first culture. If we wish to leverage open source tools in our work, we need to build team expertise in interacting with open source communities, deploying their technology in our services, and making improvements to our service by upstream contributions to open source tools.
Create an organizational culture that embodies the values we want to support. In order to build a better, more collaborative, more inclusive network of communities of practice, we need to embody the principles we wish to help grow in other communities. This will help us build trust with organizations that are inherently skeptical of “outsourcing” responsibilities to other organizations, and will make us more effective advocates for the same kinds of practices in other communities.

There are many other issues we must tackle, but we’ve identified these as being crucial to 2i2c accomplishing its mission. Over the coming months we will continue to refine our practices and service model. As we learn more, we’ll update our understanding of 2i2c’s situation and potential for impact, and our strategy for accomplishing our mission.

We hope that this framing helps others understand our position relative to the communities we wish to serve, and the impact we wish to have. We welcome any feedback or suggestions about how we can refine these ideas to better-serve our communities.

Appendix: Future growth #

Many people also ask what the growth model looks like for 2i2c. Our pricing strategy above should include our goal of building a financial buffer in order to boost the financial resiliency of the organization, and to provide us room to grow if there is an opportunity to expand our service.

While we want to focus our efforts on the strategic goals described above, here is how I think 2i2c could grow its operations and expand its impact as a next step. These are initiatives for which we might seek grant funding to experiment with sustainable and scalable models.

Create a software development team. Right now our technical efforts are focused on integrating pre-existing tools and services, and managing these services in the cloud. However there are many opportunities to create and enhance the ecosystem of tools that is out there. Doing dedicated software development is very different from Site Reliability Engineering, and we will need to build our financial and staffing capacity to dedicate team members to this work.
Expand the kinds of cloud services we offer. Currently we focus our efforts around providing research and analysis platforms in the cloud. These are often connected to other cloud resources (like datasets) but we do not manage them ourselves. We believe there’s an opportunity to strategically grow the services we provide - most notably in the area of cloud data management and migration. However, this is beyond our current capacity and we will need to grow more before moving in this direction.
Create an open source support operation. There are many things that open source communities need beyond technical improvements. Providing community, operational, and strategic support is another way that 2i2c could use its resources and experience to benefit open source communities that we work with. However, we need to better-understand how to do this in a way that is sustainable for us, and that respects open source communities as independent organizations with their own mission, strategy, and expectations.
Create a cloud infrastructure training operation. Many communities we work with aspire to run their own cloud infrastructure. They wish to work with a trusted partner like 2i2c in the short-term, but in the long term they want to build their own capacity in running similar services. 2i2c’s cloud services are a unique opportunity for others to get hands-on training to grow their skills. Perhaps there is a way for us to share this experience with members from other communities in a way that they can then bring back with them to other organizations. There will be an inherent tension between this and our sustainability model from ongoing services, but we believe it is an important opportunity to explore since it aligns with our values of community agency and choice.

Acknowledgments #

Support for organizational and strategic work like this is provided by CZI.

Celebrating our progress in Q1 2022

Mon, 04 Apr 2022 00:00:00 +0000

Quarter 1 of 2022 just wrapped up, and the 2i2c team has been busy making improvements across our infrastructure, organization, and operations. This is a quick post to celebrate the work we’ve done over the past three months, and to briefly share what we’re working on next.

These are the main highlights from this quarter - if you’d like to check out more of the work that we’ve done, see:

Infrastructure improvements #

This quarter we did a deep dive into a number of core infrastructure improvements for our Managed JupyterHubs Service. Here are a few highlights:

Infrastructure reliability and efficiency. We improved the resiliency, reliability, and efficiency of our deployment infrastructure. For example, we refactored our hub configuration so that each community is better-able to track it, we implemented validation steps to ensure that we don’t accidentally push incorrect config to the hubs, and we’ve significantly improved our CI/CD pipeline to push deployments out to our hubs more efficiently.
Automatic deployments to commercial cloud. With the ICESat hackweek as a test-case for AWS, we’ve finished automating the deployment of clusters and hubs to each major commercial cloud. (there’s not a specific issue for this as it has been a multi-month effort over many PRs and issues!)
CILogon authentication. CILogon is a non-profit organization that provides “single-sign on” authentication services for the same communities that 2i2c serves. We’ve partnered with them to prototype using CILogon for 2i2c’s hubs, which should make it much easier for communities to user their own institutional sign-ons.

Communities we’ve served and lessons learned #

As described in our Managed Hub Services strategy, our goals for this phase of our organization are to balance serving communities of practice and learning where we can improve our infrastructure and practices. With that in mind, here are a few highlights of communities we’ve served, and what we’ve learned from it:

We grew a hub for the University of Toronto to around 4000 monthly users. This has taught us a lot about where our support and operations can and cannot scale, and where we have gaps in our sustainability / pricing model.
We deployed CILogon on a hub for a class at Australian National University. This gives us an opportunity to work out any UX issues and improvements to be made before a deeper CILogon integration.
We deployed a dedicated database per user for a databases course at UT Austin. This is helping us learn more about how to pair slightly more customized per-user infrastructure with our standard hub setups, as well as how our Right to Replicate model could be followed for more complex setups like a database.
We ran an event hub for the ICESat2 HackWeek at the University of Washington. This helped refine our infrastructure and expertise with AWS, as well as improved our event “ready mode” practices.
We deployed a new hub for the LEAP project. This has given us an opportunity to prototype new processes for pass-through cloud costs to simplify our deployments.

Organizational improvements

Beyond our technical and community impact work, we’ve made a lot of significant organizational improvements as well.

We designed a new role in Product and Community Management. We’re excited for this new hire to spearhead efforts in guiding and developing relationships with the communities we serve, as well as guiding and collaborating with our engineering team in developing our services.
We designed a new Project Manager role. Our engineering team had been operating as a largely autonomous and independent group, but we’ve realized that we would benefit from someone to help coordinate our actions and plans, especially as we balance more operations/support issues in addition to new development. This new role is an experiment at growing this capacity within our team, in the hopes that we can dedicate a team member to it in the future.

What’s next #

We are still working out our major priorities for the oncoming quarter, but have a few major projects in the works that we’re hopeful to make progress on quickly. Here are a few major examples:

Improve our process and operations around supporting our users. We are discussing first- and second-line support processes to make our team more responsive and effective at resolving incidents.
Improve our invoicing and contracting process. We are discussing how to reduce toil associated with invoicing in order to make this practice more reliable and efficient, along with our fiscal sponsor Code for Science and Society.
Improving our reporting and monitoring infrastructure. We’d like to boost our ability to monitor activity on each of our hubs in order to identify when something abnormal is happening and get ahead of any potential problems (e.g., to avoid unintentionally large cloud bills). We’d also like to improve our usage reporting to more create more accurate cloud bills for hubs running on multi-tenant clusters.

There is a lot more planned for 2i2c, and if you’re curious to see what we’re up to, we invite you to check out our team compass to learn about our practices, and watch our activity on GitHub to see our work.

Many thanks to the amazing 2i2c team, and the multiple open source and scholarly communities that we collaborate with to make all of this possible. In addition, we are grateful to our funders for making this possible. We are looking forward to Q2! 🎉

Acknowledgments #

Support for organizational and strategic work like this is provided by a grant from CZI and fees from our Member Communities.

2i2c’s first year, part 1: exploring Jupyter services.

Tue, 25 Jan 2022 00:00:00 +0000

Now that 2021 has come to an end, this marks the end of 2i2c’s first year of operations. In this year we have grown, experimented, and accomplished a lot - we have also faced challenges and learned as a team. Our primary goal in 2021 was to build a strong foundation for 2i2c.

To reflect on our work thus far, we’re writing three blog posts that describe progress in major areas of work towards this foundation for 2i2c. These three areas are:

Creating services for use-cases - these are our first managed infrastructure offerings for communities in research and education.
Developing cloud infrastructure and tools - this is the technical backbone that makes these services possible, built entirely on open source tools.
Building an organizational foundation - this is the creation of our team structures, processes, and culture that help us carry out our mission.

This first post will focus on services that we’ve created in our first year.

Ultimately, 2i2c’s mission is to facilitate use-cases in research and education via open source development and services. Throughout 2021 we ran several pilots to learn more about the needs of communities in research and education, and how we could build sustainable services that meet these needs. Here are some highlights for each major use-case we have targeted thus far:

Educational community hubs #

A primary use-case that 2i2c seeks to enable is collaborative, distributed educational spaces for learning with data. In 2021 we ran several pilots with educational communities:

A university-wide hub for the University of Toronto. This hub is used in a variety of classes throughout the university, and is made freely available to anyone with a UofT account. We hope to repeat this model for other university-wide communities, and have learned a lot about the challenges of working with particularly large educational communities.

Several hubs for community colleges across California. In partnership with UC Berkeley and CloudBank, we’ve run several hubs for nearly a dozen small community colleges teaching the Data 8 curriculum for their students. These hubs are lightweight and offer standardized environments for their students to use, in order to lower the cost of deploying and maintaining the hubs over time.

What we learned #

The organizational context around educational use-cases is different from research communities. Compared with research groups, educucational groups have more variance in their size (classes as small as 10 people and as large as 500) and think about cost by the student, not as a lump sum. This means that our initial hub-based pricing model may not map cleanly onto educational contexts, and we need to improve the match of scalability and price to these communities.

Moreover, we’ve learned that for these communities, navigating all of the open source tools that are available for pedagogy is confusing! Everybody wants auto-grading but it’s unclear what is the “right tool for the job”. Tools like nbgitpuller have heavy use at “power universities” like UC Berkeley, but many others don’t know that it exists! We will need to invest more time into building guides and documentation that help others leverage these tools.

Research in the cloud #

In addition to educational use-cases, we ran several pilots for research communities in order to leverage cloud infrastructure for scaling their work or collaborating more effectively.

We migrated Pangeo’s cloud infrastructure to be run via 2i2c. The Pangeo Community had been operating and developing their own JupyterHub for several years, but were looking for another organization to provide more reliable/sustained operations and support for their Pangeo Cloud Service. This year we migrated the service to run off of 2i2c’s deployment infrastructure.

A scalable cloud hub for a SWOT satellite team. The MEOM research group at Grenoble is doing work with the NASA SWOT satellite project. However, the datasets generated from this project are huge, and only storable via the cloud. We’ve set up a JupyterHub to provide cloud-based access to this data, running a Pangeo-like environment.

What we learned #

Research communities tend to have more usecase-specific needs than educational ones. While introductory courses in data science tend to be similar across institutions, research needs are much more unique to the problem and team at hand. Moreover, they tend to want infrastructure that runs via institutional cloud accounts. This is possible due to the flexible nature of Jupyter and JupyterHub, but brings extra challenges in bureacracy and access permissions, given that 2i2c engineers usually are not members of these organizations already.

Additionally, many research use-cases are based around the location of the data. This is because data is the hardest thing to move from cloud to cloud. For this reason, it’s important to bring interactive sessions to the data. Jupyter’s ecosystem makes this possible, but we’d like to do more to make this easier. For example, users should be able to launch interactive sessions across many clusters or datacenters, regardless of their hub’s location. They should also have more customized control over their environments.

Distributed events and hackweeks #

Finally, we had a special use-case that is a hybrid of the previous two examples. Events and hackweeks are time-bounded, focused on learning and doing, but often with research workflows and communities. They also tend to be more globally-distributed than a traditional research or education community. See below for a few examples:

The Coastal Ocean Environment Summer School in Ghana workshop is a summer school run in Ghana to share skills and workflows with researchers across Africa. It builds on workflows in the Pangeo ecosystem, and teaches attendees how to work with cloud-native data. By using a cloud-based JupyterHub for their workshop, attendees had direct access to data and compute in a way that would have been much more difficult to set up on local infrastructure.

The PaleoHackWeek hackathons were two hackathons around paleo/climate data workflows. They brought together a distributed community and use the 2i2c hub to facilitate the use of the paleoclim package throughout the hackweek.

OpenScapes champions open practices in environmental science to help uncover data-driven solutions faster. They have several cohorts of scientific trainees learning more about adopting open and collaborative workflows, and use a hub managed by 2i2c to provide remote learning environments for their workshops.

What we learned #

Initially we assumed that event-based hubs would be similar to educational use-cases. However, in practice they are more complex. First, event organizers have ever-changing communities of users on their hubs. They often want to provide access to many new users in each event, and expect only a small subset to continue using the infrastructure after an event. Finally, one-off events also generate spikes in activity that are more intense than educational use-cases, and warrant more dedicated attention from our engineers.

Challenges that we faced #

As we piloted these use-cases, we also ran into a few challenges in designing services around our infrastructure.

Balancing standard deployments, custom deployments, and scalability. Our goal is to centralize our deployment and operations infrastructure in order to reduce the maintenance burden for each community hub. This will allow us to gain economies of scale and lower costs. However, there’s an inherent tension between scalability and customizability - in order to scale we must standardize the infrastructure we offer, but communities often want environments that are unique to their use-case.

Turning services into sustainability. We’ve learned that charging money for things does not come naturally to us. We’ve spent many years building technology and services that were offered freely to the public. This was easy to provide, but difficult to sustain. 2i2c’s goal is to offer similar services, but with a sustainability model that allows us to employ team members to do this work, and to grow in the future. Understanding how to pair the right technology, use-case, and service with a particular pricing model is an area where we must learn more.

What’s next #

Piloting all of these services taught us a lot about the similarities and differences between each use case. Towards the end of 2021, this led to 2i2c’s first collection of “service offerings” for a Managed Jupyter Service. This includes a 2-by-2 matrix of service options, and a specific price point for each. We have begun offering this service to communities, in order to learn more about the model, the pricing, and any changes that should be made. In early 2022, we hope to expand this group to more communities so that we can test this model’s ability to scale.

Our next post in this series will focus on the underlying technology that powers the services described above, stay tuned for more!

Community update Q3 2021: A new fiscal sponsor, improving our infrastructure, nearing an alpha launch.

Sun, 10 Oct 2021 00:00:00 +0000

This is a (roughly) quarterly update for the 2i2c community, with the goal of providing transparency about what we’ve been up to, sharing what we are working on and where we have struggled, and discussing what we’re up to next. In addition, almost all of the work we do is public and discoverable across our GitHub repositories, and is tracked by GitHub issues. Here’s a list of issues we’ve closed in ~Q3.

It is amazing how quickly 4 months goes by when you’re building an organization from scratch! It seems like only a few weeks ago that we were recapping the beginning of the year in our last community update. Since then, we have been hard at work to make 2i2c’s organizational and infrastructure more robust and sustainable.

There are several major strategic areas where 2i2c aims to have impact, and we’ve split this community update along each of these major areas below. We’ll cover major highlights, challenges we’ve faced, and where we’re going next.

Highlights #

Managed JupyterHub Service #

Our Managed JupyterHub Service will be a sustainable, scalable, and participatory service to provide cloud-based DevOps around JupyterHub for communities of practice in research and education. For the past several months, we have been running individual JupyterHubs for many organizations as a pilot, in order to learn more about the challenges we’ll face, and give ourselves an opportunity to build centralized infrastructure around the service.

We focused on a few major areas for work, outlined below:

Automation across cloud providers. We wish to serve communities that run on any of the major commercial cloud providers. We can standardize some of our infrastructure through abstractions like Kubernetes, but must still create cloud-specific deployment infrastructure as well (that Kubernetes cluster has to come from somewhere first!). In the last four months we’ve worked on automating Kubernetes and JupyterHub deployments on AWS as well as Azure to complement our Google Cloud deployments. We would soon like to run more hubs on this infrastructure to test how well it scales.
Monitoring and reporting infrastructure. We have worked on the JupyterHub grafana-dashboards project to improve dashboarding around JupyterHub deployments in general, and will soon automatically deploy Grafana dashboards for our hubs so that communities have insight into what is going on in their hubs.
User environment management. We want communities to have control over the environments that are available on their hubs. We also want to encourage that our communities follow community standards for reproducible environments that can be re-used elsewhere. For this reason, we’ve improved the repo2docker GitHub action to work with more image registries, and created a 2i2c user image template repository for users to re-use for their hubs. See the User Environment docs for more information.
Support and collaboration roles. In addition to technology changes, we have developed an alpha-level support and collaboration model for the communities we serve. Most relevant for our communities is the community representative role, who acts as the main point of contact with 2i2c engineers, and leads administrators on the hub to guide its customization for the community it serves. See the user roles documentation for more information. We have also begun prototyping a FreshDesk support model and team processes around monitoring our support channels and responding to requests and incidents.

Pangeo #

We are working with the Pangeo Community to migrate the Pangeo JupyterHub deployments to utilize 2i2c’s centralized infrastructure, with the goal of 2i2c taking over the development and operation of Pangeo hubs moving forward. We have spent the last few months re-creating the Pangeo hub environment from scratch on a new cloud project controlled by Columbia University, and are nearly ready to begin migration from the “old” Pangeo hub to the new one. After this, we will focus our attention on re-creating the Pangeo BinderHub and AWS hub. Follow along with this work in this GitHub Project.

Executable Books / Jupyter Book #

We are nearing the final year of a grant from the Sloan Foundation to support development on the Executable Books Project. As such, we have begun shifting our attention to create a strategy for sustaining the project’s community beyond this grant. In the coming months we plan on prioritizing improving our documentation (both for users and developers), as well as improving the general maintainability and quality of our codebases.

JupyterHub community support #

We recently collaborated with the JupyterHub community to apply for two CZI EOSS awards. Last month, we were notified that our application to support a Community Strategic Lead was funded! This role will fund Sarah Gibson’s time to focus some of her thinking on building community structures and dynamics that are inclusive and sustainable. We’ll update with more information as this project starts moving.

Organizational foundations #

Finally, in addition to our major development and projects, we have also made a lot of progress at an organizational level.

We began a fiscal sponsorship with Code for Science and Society. This provides a new organizational and legal home for 2i2c after spending nearly a year receiving critical strategic and start-up support from our previous host, ICSI. We are excited to work with CS&S to create the business infrastructure that will power our managed JupyterHubs service.

The 2i2c team has also been improving our team planning and coordination processes, so that we can more effectively execute on our mission. As a distributed team, we have the challenge of building processes for team communication, coordination, and planning that are distributed and asynchronous.

We have improved our organization-wide documentation in order to make it easier to navigate between 2i2c’s various sources of information. We hope that this provides more transparency into what 2i2c is up to and how it is structured, and that it allows us to build more connections between our projects and the broader community. Check out the new documentation landing site at docs.2i2c.org.

What’s next #

In addition to making iterative improvements around the projects described above, we have a few major goals for the final quarter of 2021.

We wish to polish and finish the alpha-level deployment infrastructure across each of the major cloud providers. Our end goal is to be able to deploy a Kubernetes cluster and a JupyterHub on that cluster with minimal human effort. This will allow us to serve more communities with fewer human effort per community. This includes migrating JupyterHubs for key communities, such as the Pangeo project.

We wish to define and begin an alpha-level service. This means defining a few specific “products” that 2i2c can offer, as well as a pricing model for each product. Our goals will still be to learn about how this service should evolve, but we’ll have a bit more specificity about ways that others can engage with us. It will also be an opportunity to scale modestly so that we can improve our infrastructure as-needed.

We wish to define a longer-term vision for our Managed JupyterHubs Service, and create a plan for new workflows that we wish to enable via new development. We wish to use our experience in managing JupyterHubs to understand where new open source development would significantly help the communities we serve, and will shift our attention in this direction once we have finished the early stages of making our infrastructure more robust.

Stay tuned for some more updates about this service, coming soon!

Challenges we’ve faced #

Finally, we want to recognize that there are plenty of challenges along with successes when it comes to having impact! Here are a few of the unexpected challenges that we’ve faced along the way.

Developing and operating multi-cloud infrastructure is a complex process. Automating the creation and operation of Kubernetes clusters across each of the major cloud providers has proven to be a lot of work! While it has been necessary to use our pre-existing hubs to provide the opportunity to build this infrastructure, it is a challenge to both build and operate infrastructure at the same time. This has lengthened our timelines for when we expect the alpha version of the service to be ready.

Coordinating among a distributed team takes time. Each of our team members has been used to working asynchronously with open source communities for many years, and we under-estimated how complex it would be to port this workflow to a distributed team that must coordinate with each other much more closely on a daily basis. Deciding what to work on, how to work on it, and how to help one another take a lot of work when you only share a few hours awake at the same time. For this reason, we have spent a lot of extra energy in our first months as a team to work out team processes for coordination and collaboration.

How to design cloud services as a non-profit? Finally, we’ve learned that there is no clear roadmap for how to build a sustainable cloud service like the one we describe here. 2i2c is somewhere in-between a traditional SaaS company (which tends to provide access to proprietary technology) and a bespoke consultancy (which is more hands-on and custom than we are). Our best explanation for what 2i2c wishes to do is “Open source operations/dev-ops as a service”, which doesn’t quite fit into either category. This means that we have more “unknown unknowns” about what kind of service and pricing model we should develop. We hope to begin answering some of these questions in our alpha-level service launch. Stay tuned for more to come!

Get involved #

Thanks for your interest in 2i2c and the mission that it hopes to accomplish. We hope that this update has been informative and interesting, and we welcome feedback about any of the information provided. If you’d like to help with any of the challenges or efforts described above, we welcome all forms of collaboration, and are happy to chat. Please drop us a line at hello@2i2c.org or contact us about joining our Slack community. Thanks for reading!

Pilot hubs, new collaborations, and new team members - A six month update

Tue, 25 May 2021 00:00:00 +0000

It has been about six months since 2i2c first began operations (after receiving funding from the Chan Zuckerberg Initiative). In that time we’ve made progress along several directions, and wish to use this blog post to provide updates about the ways in which 2i2c has evolved over the first months of its existence.

Below are a few major updates from the 2i2c community - as always, if you want to learn more about 2i2c, keep an eye on our blog or subscribe to the 2i2c mailing list.

Early pilot JupyterHub infrastructure #

First off - we have been making progress building out our JupyterHub deployment infrastructure for 2i2c. One of our major organizational goals is to build a sustainable service managing open source cloud infrastructure for interactive computing. This service will provide hosted, customized JupyterHubs for communities of practice in research and education. They’ll be built entirely with open source tools that are community-driven, and that respect the community’s Right to Replicate.

In order to accomplish this, 2i2c is running several pilots with partners and interested organizations, supported by our funding from CZI, as well as from the JROST rapid response fund. These pilots are meant to be learning opportunities to understand what kind of infrastructure and service it needs to build moving forward.

The documentation for our pilot hubs infrastructure contains information about our deployments and infrastructure. It is served from this 2i2c-org/infrastructure repository, a centralized location for configuring and deploying a federated network of JupyterHubs. Each JupyterHub is independent of one another, and could be spun out from the centralized repository with minimal extra work, giving hub users the ability to replicate their infrastructure, with or without 2i2c. We will continue refining the code in this repository as we learn more from our hub infrastructure pilots.

JupyterHub for geospatial analytics - A collaboration with Pangeo #

As originally announced on the Pangeo blog, 2i2c is forging a collaboration with the Pangeo project around operating and developing cloud infrastructure for large-scale geospatial analytics! This collaboration is funded through a grant from the Moore Foundation (via Pangeo investigator Ryan Abernathey).

Over the coming months, 2i2c plans to assume operation of infrastructure underlying the Pangeo project, allowing the Pangeo team to focus their efforts on their core scientific and development missions. Because Pangeo’s infrastructure is already running on a fully open source stack with JupyterHub, our first step will simply be to shift control over this infrastructure to 2i2c engineers. We don’t anticipate needing to make major changes to their infrastructure and deployments (part of the benefits of using open, modular tools).

Once this is complete, we’ll next shift our attention to some new areas of development that support use-cases in the Pangeo community (and in the scientific community more broadly). There’s a lot of progress that we imagine making - such as supporting sharing pipelines via the Pangeo Gallery or improving tools for scalable computing with Dask Gateway. We’ll provide updates as we formally begin this collaboration and hash out a plan for our next steps.

JupyterHub for education - A collaboration with CloudBank and UC Berkeley #

In addition, we’ve begun a partnership with the UC Berkeley Data Science in Undergraduate Studies program, as well as CloudBank. This collaboration aims to provide hosted JupyterHub infrastructure for community colleges across the state of California. It is an attempt at providing vendor-agnostic and open-source infrastructure to several institutions who would otherwise not be able to deploy this infrastructure on their own.

2i2c will provide the deployment and configuration architecture for this collaboration, working with Sean Morris in operating this educational infrastructure. All of the cloud infrastructure for this pilot will be funded via CloudBank. We will begin by offering environments that are modeled after the Data 8 course at UC Berkeley. This is part of an effort to build a community of practice around Data Science education using open source tools.

New team members #

We’ve also welcomed two new members to the 2i2c core team! 🎉

These individuals will both work towards 2i2c’s major projects, and collaborate together on running our 2i2c Pilot Hub infrastructure. Here’s a bit about each new team member.

Damián Avila. Damián has been a Jupyter core team member for many years now, and has done work across many different parts of the PyData stack (in particular, Jupyter, Bokeh, RISE, and Nikola). Damián will focus his efforts on supporting JupyterHub infrastructure for the Pangeo project, as well as development across the Executable Books Project
Sarah Gibson. Sarah will join 2i2c in June, after spending several years as a Research Software Engineer at the Turing Institute. She has also been involved with the Turing Way for many years. Sarah will focus her efforts on JupyterHub development and operations for the Pangeo community.

Governance and a code of conduct #

Finally, while it’s easy to get lost in technology and collaborations, 2i2c has also made important steps towards defining a stable and transparent organizational model moving forward. 2i2c now has a Steering Council and an early organizational structure. In addition, we’ve defined a one-year bootstrap strategy that we’ll use to guide our path in the first year of 2i2c’s existence.

Finally, one of the first acts of the Steering Council has been to adopt a Code of Conduct. This is a set of guidelines, and a process for resolving incidents, that makes our community more inclusive, equitable, and enjoyable for all. Creating a Code of Conduct is a crucial part of defining our organizational and community culture, and we’re excited to have some explicit guidelines to support our interactions in the future!

Keep in touch #

Now that we are improving our organizational foundation, we will try to post more frequent updates and discussions about what we are up to. We hope for 2i2c to be a model organization in participatory, collaborative, transparent operations, and we look forward to working with you all in the future on this journey.

Hello World

Tue, 10 Nov 2020 00:00:00 +0000

👋 hey everyone!

We’d like to announce the creation of a new non-profit organization¹ that we call 2i2c.

2i2c stands for The International Interactive Computing Collaboration. It is a non-profit dedicated to making open tools for interactive computing more accessible and more powerful for the research and education communities.

This is a short post about why we created 2i2c, what we hope that it will do, and what we are up to next. If you’d prefer to watch a video instead of read a blog post, check out this talk about 2i2c at JupyterCon 2020:

Why create 2i2c? #

The founding team of 2i2c has spent the last several years running projects that use interactive computing for research and education, including bringing data science education to thousands of students, connecting geospatial researchers with large datasets and computational resources in the cloud, and providing federated online environment hubs to schools across Canada, to name a few.

Over time we realized that, while infrastructure for interactive computing could be a huge benefit to research and education, it also required a fair amount of expertise to configure, deploy, and develop. We wished for other organizations to enjoy the same success that we had found, but learned that for many, deploying their own infrastructure was a non-starter to adoption. Instead, many were turning to proprietary or vendor-specific tooling.

We created 2i2c so that these organizations can use entirely open-source tools without hiring and training their own dev-ops and infrastructure talent, and so that development and support of open tools for interactive computing continues to represent the interests of research and education.

Why a non-profit? #

It may sound strange to create a non-profit initiative when there are many VC-funded startups and large tech companies offering notebook services these days. However, we think that a non-profit organization is the right approach to balance the interests of all the stakeholders that we wish to serve. We hope that 2i2c will be a partner to:

Research and educational communities, who can rely on 2i2c to provide them cutting-edge infrastructure for interactive computing that is 💯 open source.
Researchers and educators who need development, who can rely on 2i2c as a collaborator that offers development and expertise in open-source infrastructure to push the cutting edge of interactive computing in the cloud.
Open source communities, who can count on 2i2c support and grow the communities that underlie the tools that we deploy.
Cloud providers, who wish to help the research and educational community via their infrastructure.
Supporters of open source who wish to support interactive computing for research and education via a non-profit dedicated to exactly this mission.

As a non-profit initiative, 2i2c is dedicated to supporting an ecosystem of tools and stakeholders across the open source community, and to ensuring that those tools are well-suited for research and education. We believe strongly in mission-driven work, and non-profit status will ensure that the work that we do is always aligned with our mission and values.

What are we going to do? #

With all of that in mind, what is 2i2c actually going to do? We are still working out the details, but here’s a rough picture:

Offer hosted interactive computing environments on cloud infrastructure. These will be entirely open source and vendor-neutral, and customizable for the communities that are using them. They’ll be offered either as a fee-for-service model and/or subsidized through grants and donations. We wish to build upon the success of JupyterHub as a gateway to computational resources and environments, learning environments, and communities of users. For more information about the vision and values of our hosted infrastructure, see the 2i2c Right to Replicate document.

Provide collaboration and development for interactive computing in research and education. Beyond providing hub infrastructure, there are many ways in which solving problems in research and education can lead to better tools, infrastructure, and workflows in the open source community. For example - how can we generalize a community’s solution to scalable computing so that it can be useful for other use-cases as well? We hope that 2i2c can be an aggregator and integrator of many perspectives in research and education, and build tools that are maximally useful across communities.

Provide core development and community support for open source projects that we use. While many organizations use Jupyter technology in their projects, it is also crucial that they give back to those tools in order to keep the ecosystem healthy. As a mission-driven non-profit, 2i2c has a core goal in not only deploying and customizing open source tools, but also providing core support for them.

Next steps #

2i2c is a young organization, but we already have a few exciting ideas to work towards in the coming months. Here’s an idea of what we’ll be up to next.

Our first step is to understand how interactive computing can best-serve the research and education communities. We know that interactive computing tools work in our specific use-cases, but how do these generalize to other domains or types of organizations? Moreover, how can we improve the open source tools to make them even more customizable for each community?

We’ll begin answering this question through several focused JupyterHub pilots aimed at different use-cases across research and education:

A “Hubs for All” pilot for education. This will focus on making standard educational environments via JupyterHub accessible to a broad range of educational institutions, in particular, traditionally under-resourced institutions such as community colleges, state colleges, and HBCUs.
Research-focused hubs for earth analytics. Research requires more complex and customized infrastructure than education, and so we are piloting hub infrastructure for a few specific research communities to understand how we can meet their needs. We will use this to better-understand how 2i2c and support a diverse research community in the future.
JupyterHub development for the Pangeo Project. Pangeo has been a collaborator of 2i2c since day one, and we love its vision for large-scale earth analytics via cloud infrastructure. We plan to work closely with Pangeo leadership to work on early development of tools aimed at facilitating scalable computing for research on Pangeo infrastructure. More information about this to come soon.

We’ll also use these pilots to develop an organizational sustainability model around this infrastructure. It takes expertise to manage and develop cloud infrastructure, and 2i2c wishes to pay market rates for its engineers. How can 2i2c offer services and development in a way that is sustainable both for itself and for other institutions? We’ll use these pilots to create a plan for both sustaining and scaling 2i2c’s ability to serve institutions and teams.

What do we need? #

As a non-profit initiative, 2i2c has a long road ahead of it. We cannot take venture capital funding, so we have to raise funds the old-fashioned way. This means that:

If you are at an organization interested in purchasing hosted JupyterHub environments from 2i2c, please send us an email .
If you are a funder or are otherwise interested in supporting 2i2c with a donation or grant, please send us an email .
If you have experience in sustaining service-based organizations such as 2i2c, and wish to offer your expertise or guidance, please send us an email .

Moreover, we’ll likely be hiring in the coming months for open source engineers, with a focus on data science, interactive computing, and the cloud. If you’re interested in working for an organization like 2i2c, please send us an email , and keep an eye on this space as we will post more information soon.

Get involved #

If you’d like to get involved with the 2i2c team and community, we recommend two pathways for doing so.

Subscribe to our mailing list. We’ll use this to periodically send updates about what 2i2c is up to.

Join the 2i2c Slack. We use Slack for team conversation, as well as for real-time conversations about tools, practices, and ideas around interactive computing for research and education.

You can also say hello to us on Twitter at @2i2c_org.

Wrapping up #

We are excited about 2i2c, and believe that it is the right kind of organization to create to support research and education, as well as open source communities in interactive computing. The road ahead is a difficult one, but we’re confident that we will hit our stride quickly. Stay tuned for more updates to come.

Acknowledgements #

Support for organizational and strategic work like this is provided by a grant from CZI.

EDIT 08-2021: We are now a fiscally-sponsored project of Code for Science and Society and ICSI is no longer our host organization. We are leaving this section here for archival purposes, and to give credit to ICSI for their help in launching 2i2c. 2i2c’s host organization is the International Computer Science Institute, a 501(c)(3) charitable organization. We thank ICSI for its collaboration and leadership in launching 2i2c! ↩︎

Report | 2i2c

Report from the Jupyter Security Working Group security tooling sprint

Acknowledgements #

2024 impact report: new team structure, new funding, and growth in our network

Organizational updates #

Community impact #

Open source technology enhancements #

Looking to next year #

Acknowledgments #

Community Update: October 2023

Rough Numbers #

Organization Updates #

Improving Our Quarterly Sprints and Goals #

Three New Jobs Posted #

Product Strategy Work #

Continued Onboarding of Communities in Our Catalyst Project Collaboration #

Partnerships and Impact #

What We’ve Learned and What Stood Out #

Open organizational report: Strengths and challenges for 2i2c's team

Acknowledgments #

Community update Q3 2023: Service growth and growing pains

How has our service evolved over the past few months? #

New partnerships and service growth #

Financial picture #

How service growth can lead to team stress #

Growing the complexity of our team to match the complexity of our service #

We need a dedicated product functionality #

We need a more structured system of work and execution #

We need more capacity to support our team and its individuals #

We need more engineering capacity #

What’s up next for our team? #

Acknowledgments #

Community update Q1 2023: Growing our community partner network and our team

New community partnerships #

Service improvements #

We simplified our authentication workflow with CILogon #

We exposed user activity dashboards in JupyterHub so communities know how many people are using the service #

We made our support process more structured with a new support widget #

We upgraded Kubernetes across all of our AWS clusters #

We streamlined our hub uptime checks to be more efficient #

Organizational updates #

We defined organizational principles around engaging in healthy ways with open source communities #

We responded to a NASA RFI on “Scientific Data and Computing Architecture to Support Open Science” #

We added two new team members and a new team role #

OpenScapes wrote up a blog post about our cloud service collaboration #

Onward to Q2 #

Acknowledgments #

2022 in review: growing our partner communities and expanding our operations

Our major goals in 2022 #

Highlights from 2022 #

We grew the number of our partner communities #

We grew our revenue from community partnerships #

We got grant funding to serve communities in Latin America and Africa #

We improved our continuous integration and deployment system #

We defined a Shared Responsibility Model #

We defined a formal Incident Response process #

We expanded our service offerings to include community and workflow guidance #

We began a collaboration with GESIS to develop environment building in JupyterHub #

We helped maintain several upstream open source projects #

We refined our organizational strategy #

Things to accomplish in 2023 #

Sharpen our Shared Responsibility Model #

Grow our community support operations #

Define a cost recovery model that balances sustainability and accessibility #

Incorporate a dedicated software development practice that aligns with open values #

Acknowledgments #

Celebrating our progress in Q3 2022

Community impact #

Organization wide updates #

Service improvements #

Where we’re focusing next #

Thanks #

One year later: an update of 2i2c's mission, strategy, and impact

What did we miss with our original strategy? #

Refining our strategy #

Context: Our key stakeholders and the impact we wish to have #

Assumptions we make about our stakeholders #

Open source tools have boosted standardization and collaboration across communities #

…but a local workflow with open source tools is inaccessible or underpowered for many people #

Managed cloud services can solve the accessibility problem for many workflows and tools #