Thank you.
I'm going to try to speak loudly, which hopefully will be OK, and hopefully not too bad for people online as well.
So, yeah, so thanks for the opportunity to be here.
In this endeavor, I'm going to talk about research software engineering, kind of in one stream,
and then what's been going on at the University of Illinois or Banish campaign with regard to research software engineering kind of at the same time.
So, I guess first, just how many people know research software engineering?
OK, this is probably an unusual room here.
That's hard to tell.
So, wait, now I feel like maybe I should just skip the first six or seven slides.
Well, we'll try and see what happens.
So, maybe there will be something that you don't know.
OK, so first, I'm going to start off with basically just defining research software, which I think probably everybody here has some idea.
I don't know how many people have a formal definition.
And so, in this group that more end-grimbeder led with a few of us over a couple of years ending in 2021,
we tried to define this through a discussion.
And the definition that we came up with is that research software has a bunch of different forms,
but it's things that were created during the research process or for research purpose.
And there are other pieces of software that are used in research, like operating systems and libraries
and dependencies and package management systems and other things that are that weren't created for research,
and they're not really intended for research.
And so, we said that those should be called software in research and not research software.
And this varies between disciplines.
Sorry if you heard me say this yesterday, but if you are, I'll make a different example.
If you are a computational linguist, and you have some code that you've written in Fortran,
let's make this a really strange example.
You need a compiler in order to actually run that code.
And so, the compiler that you're using is not research software for you as a linguist.
But, on the other hand, if you're a computer scientist who is studying Fortran compilers,
maybe there's such a thing still.
That same compiler, if you would be research software.
So, it's a context question.
We can argue about this definition for a while, and in this process, we did argue about it for about 18 months.
And we came up with this as the best thing that we could agree on.
It's not necessarily the right thing.
It's not necessarily what any one person thought was the right thing, but it's what we got to.
And lately when I've been saying this in talks, people want to have this discussion again.
They say, can we have a community group that goes through and comes up with a better definition?
And my answer is, sure, let's try it and see.
I don't think we'll come up with a better definition, but if anybody wants to try it,
I think it's worth doing, and maybe we will.
Okay.
So, research software then typically comes from research.
This has happened from the start of computing, more or less.
Software started appearing around 1948 as best I know, somebody can correct me if they have a better definition of better term.
Research software started appearing in the early 1950s, just a few years later, and it was initially weather software as far as I can tell.
And then another roughly 20 years more or less until we got to software engineering and starting to think about how do we do this in a reasonably professional way as a discipline.
The people that are writing research software who are researchers, sometimes faculty, sometimes other people generally don't know good software practices still.
And we have software engineers who really do know good research, sorry, good software practices, but they don't understand the context of research.
And so you get situations where software engineer comes in and they say, please write down the requirements and then I will build software that meets those requirements.
You say, I don't know the requirements, this is research, and so there's that kind of question about how do we go back and forth between those things.
At universities and other places where research happens, you often do have students and postdocs who initially don't know good software practices, but they kind of pick them up over time during their career.
And they get to a point where they're actually fairly decent at this sometimes, and then they leave and they go somewhere else.
Some of them stay around and they become the staff at an institution and that's great if that happens.
Sometimes they're called a postdoc for a very long time and they're not officially ever called a staff member, but that's a different problem.
And and eventually we do get to a point where we have staff who have an understanding of research as well as software engineering skills.
And I think that's where we are today.
And so one one way of looking at this is the difference between the the crafts person and the scholar where we can say that a scholar is a researcher who's really driven to understand things.
They find intellectually demanding problems, they work on them until they really understand them and then they pass on that understanding through teaching.
And then we have crafts people who are driven to create and and leave behind an artifact that really.
That that really says what they understand that that is the understanding in some sense.
This is people that feel pain when they make something that's fragile or ugly.
They prefer to make things that explain themselves although they certainly often will understand the need for documentation as well.
And this is work that requires patience and pride in doing a job well.
And so scientific software research software needs people that can understand and do both of these things.
And I just want to point out this is from blog post from 2012.
So this is not a new idea at all. This is something that's been around for almost 15 years at this point.
Sorry.
So if we then think of a research software engineer and an effort to try to define this a little bit.
We have a space which I'm going to define with two axes because we're in a research context so I can do this.
We have a software engineering axis and we have a research axis.
And so we have people that are software engineers that are really great at software engineering.
We have people who are researchers that are great at research.
We have people who are a researcher of developers who are kind of somewhere in the middle that are quite good at research but also are decent at software.
And the idea is that we have research software engineers who are really more in the middle and can do both of these things.
And and there's not a point to those kind of a cloud of which we have research software engineers.
near some or better research, some are better at software engineering, but they're pretty
good at both of these things, and that's kind of the key here.
So in 2012 in the UK I believe in Oxford, but I'm not 100% sure if I'm right about that.
There was a collaborations workshop, and breakout group at this workshop was talking about
this problem, and what they realized was that there were a lot of people that were doing
this work, that were doing research software with an understanding of research and trying
to apply good software engineering practices.
But they didn't have a common title, and in the UK they had 194 different job titles
that they found in job listings that were basically doing the same work.
So they said, let's come up with a common title, let's call this research software engineer
RSE.
They decided that there wasn't any community of these people, all these people with different
jobs and different places, different titles had felt like they were alone often, and so
they created an association that's turned into a society, and there's now other societies
and associations and other places.
And they realized that this wasn't a profession, and so they started working with their
universities to develop career paths and a formal structure by which people could be
hired and could get promotions and could be evaluated based on their work in this job.
So 13 and plus years later we have, again, this movement, this term that was born in
the UK, the UK RSE Association formed with about 50 people in 2013.
It's now a professional society, a non-profit, a dues-paying organization with 700 official
members and almost 7,000 unofficial members.
And there's other associations in a lot of other countries at this point as well.
Belgium, Germany, Netherlands, Nordic countries, Australia, New Zealand, Canada, Chile,
in the US, we have a US RSE organization that has about 3,800 members now, so we're kind
of growing a little bit behind the UK, but catching up hopefully.
There's an RSE-like group in France that's called Devlog, that actually started before
the RSE movement, and it has kind of its own history and does things all a little bit
differently, but there's a lot of overlaps as well.
And there's new associations that are forming or at least trying to form in Africa and Asia,
but primarily I would say in South Africa and India at this point, but trying to expand
beyond those as well.
And these associations then can work on local issues collectively, so in the US, working
on issues that are specific to the US, but then also we can coordinate when there's an effort
to try to work together across countries to try to change culture, to change practices
at a higher level.
So that's the RSE side of things.
If I go then on to the side at the National Center for Super Computing Applications at the
University of Illinois, what we've been doing, NCSA started in 1986, so 40 years ago,
actually like a week ago or so, it was our 40th anniversary, a professor at Illinois submitted
a proposal without being requested to, without submitting a two particular call, to create
a US high performance computing, HPC capability the next year, sorry, in 1985.
And after a lot of discussion from the funding agency they decided to seem like a good
idea and it's something they should do, and they supported five national centers that were
formed, and NCSA was one of those five.
We ended up then, I would say, catalyzing the internet and the web under this support.
In the first year we created NCSA Telnet, which was in some ways intended to help people
connect to these supercomputers.
We created HTTBD to serve websites so that people could then share the information that
they were creating.
We created Mosaic, a few years later, which was the first graphical web browser that I would
say in a lot of ways kind of started, started a lot of the web as, well, initially as a scientific
activity, and then eventually into more of a commercial activity.
And then we got to a point where there were some problems.
So the core funding that we were getting from the funding agency ramped down over a few
years.
So we had this core funding that supported everybody for 10 years, and then five years later
we didn't have any of this core funding.
And so that led to a lot of challenges in terms of what do people do and how flexible are
they and how do they work.
All of the funding became competitive, so in order to get funding we had to write proposals
and compete with people in every other university, every other lab.
Some large projects, one, some of these competitions, as well as some small projects.
But then some people left because they were unhappy with this competitive landscape, and
they were looking for other things, and we also wrote some proposals that didn't exceed,
and so then there wasn't money for everybody and some other people left.
And we ended up at a situation where we had multiple groups in software that had overlapping
skills.
We had internal competition, where different groups were creating projects that competed
with each other, so we were developing three different workflow systems.
We had internal competition on grants about who would be allowed to propose to particular
things if there was a limit.
Internal competition for collaborators, the good computational scientists in some department
would be approached by multiple groups at NCSA that wanted to collaborate with them, and
they didn't want to do that.
They just wanted to collaborate with one person and do their research.
Internal competition on staff, when we wanted to hire somebody who would hire them and
how that work, huge discrepancies on titles and salaries, because different groups were running
under different systems, and so that led to unhappy employees and inequity, and people
that couldn't necessarily move from one group to another one project to another easily.
And nuclear entry point for new collaborations, as I said, and finally no clear path
for advancement except if you were a PI and you want to grant, that was a good thing
and that would help you advance.
Oh, and no job security as well, because if you didn't win, then that was it.
Okay, so that was bad.
But then things got a little bit better, more than a little bit better.
So one of the NCSA groups at that time was called Image and Spatial Data Analysis
or ISDA, and this was kind of an isolated group that was working in this particular area
of Image and Data Processing.
And a person named Kenson McKennery, who is one of the co-authors of this talk, took over
that group, and he was frustrated with the situation and he wrote a manifesto that basically
said how things should work in his opinion.
And that included things that said basically the PI role as sacred, the PI's are the
people that should be making decisions, not some other level of management.
There should be an emphasis on job security, governance and decision making should be
as democratic as possible among the PI's at one level and among the staff at another level.
There should be an agreed upon hierarchy.
They should be agreement between the PI's about when to hire and when to expand and who
to hire.
That shouldn't be each person doing it just for their project.
And there should be agreement on which projects to pursue.
And that trust was the key for efficiency.
And that if the PI's could trust each other and could work together, then they could actually
collectively support all of the people and grow the organization.
And so some new principles then came out of this.
And the principles that we ended up with were to try to align projects and diversify funding.
So the idea is that if you have a big project, you use it to start small projects that
do some of the pieces.
And if you have small projects, you can bring them together to turn them into bigger projects.
And the bigger projects get bigger funding that's more sustainable, the small projects
have shorter-term funding, but there are ways of exploring ideas.
So thinking of these all together as a system, trying to reuse this software as much as possible,
trying to actually be a group as opposed to a set of projects.
So the PI and staff have a larger voice, a louder voice together, to have a management
committee that beats, in addition to team meetings, to bring everybody together again,
to look for opportunities for synergies and ways of leveraging the work that's happening
in one place and another place and to provide stability for staff.
So if one project ends, there are other projects that can hire the staff because they
were involved in bringing them in initially and they know who they are and they know what
they do.
To think of ourselves as collaborators and not contractors, and I think this is actually
a big deal that happens for a lot of research software engineers, but it was really, I
think a big change for us.
So if we're working with somebody that's a researcher, we want to be seen as a copi, we
want to be involved in the process of developing the ideas of doing the work.
To be co-authors on the paper that comes out, things like that, not just to be somebody
that you go to and say, can you write some code for me that does this?
And then finally, and this is a little bit repetitive, but a little different way of looking
at it, is to think of staff members as working for the institution, not for the project,
and they're hired by the institution, they're not hired by the project.
But there's one broad shared reusable continuous job ad that is reused by any project that
needs to hire people.
We don't have to have each project write their own job ad and have their own competitions.
That there's a rotating hiring committee that helps spread the load across people who have
a formal career path, and we use senior staff to do training in a formal way as well.
So vision and goals for NCSAN software, so one of them is that we support scientific discovery
through software, and this is really research discovery.
Sorry, we always have issues with using science versus research, but it's really scholarly
research.
We amplify the efforts we support.
Again, we're not just writing code, we're actually making the project better.
We're known for our staff in our software, and we're a leader in the research software
engineering movement.
We strive to lead in the areas that we undertake, and that we embody trust and responsibility.
So again, these are principles that we try to work with with different researchers.
We don't always do all of these as well as we should, but this is what we try to do.
So if we look then back at the RSE movement and how we're fitting into it, so I joined
NCSA in 2016 about 10 years ago, and I was active in the RSE movement through the UK
at that point.
And so I brought this idea, and I said, you know, I think this is what we really are.
We are RSEs.
We're not just research developers.
I think it was the title that was being used before that.
And so this gave us a sense that we at NCSA weren't alone.
There were other people that had the same problems, and we could learn from them and work
with them.
In 2018, I co-founded the U.S. RSE organization, and I was on the steering committee for
the first four years, and then Kenton McKennery from, again, from NCSA was on for the next
three years, and now we have Pinking Shen, who is also on this.
So we're trying to make sure that we have a continual role in shaping the organization.
This is the member's vote for us.
And we see ourselves as a U.S. exemplar of an RSE group.
And so I just want to point out one paper that we wrote in 2019 that was looking at the
model that we had, the model that was taking place at the University of Notre Dame, and
the model that was taking place at the University of Manchester in the UK.
And so we compared these three different models for how do these different groups work,
how do they hire people, how do they find projects, how do they support themselves.
And we think that this is, I don't know, three out of maybe, I don't know, five, eight
different models that maybe makes sense, maybe there's more, but we think people that want
to start a new group can look at these models and see what other people have done.
We also have, again, a formal career path, I know you can't read this, sorry, let me
just jump back for this second and say, the slides are at this DOI, they're also linked
from the FASTAM page about this talk, so you can look at any of these at any point.
We do have a formal structure where we are, things like responsibilities that change over
five levels from associate research program or all the way up to principal research
program, decision making that comes in a different point, supervision that comes in
at different points, kind of requirements about education and training and experience,
and then kind of preferred experience.
And so we've tried to, again, formalize this so that when people want a promotion, we
know what they need to be doing at the next level and we can see, are they able to do that
and they can see what do they need to learn as well. And this gives people also, of course,
the chance to move up over time. So getting close to the end, just to mention, we kind of have
three different formal project roles right now. We have the PI or copi or the Stewart, the
person that's responsible for this excessive approach. Their job is primarily oversight and
guidance and organization and helping staffing and being the person that's responsible for the
deliverables. This is generally something that's working like about 8% of the time. So this is kind
of like one month a year, spread out over a year. And these people can work on multiple projects at
once. And we have technical leads who are the person that's really responsible for overseeing
development, coordinating the developers, probably doing some of the development themselves,
but also training the developers, the newer people, the younger people that are coming in.
And this is often about a 10% position. And again, they can be doing multiple of these at the same
time. And then we have the core developers who are actually doing the real main development work.
Perhaps they're involved in publications and reports as well as they get somewhat more senior.
Ideally, this is 50% time. They're working on two of these at a time. In practice, we don't always
get 50% of people from a project. And so sometimes they can be working on maybe up to four
for more senior things. So maybe four projects at a time. We try to keep junior people more at
two, but again, as people go up, maybe it's a little more. I mean, we'll say 50% time ideal.
The reason we say that is that if we get full time support for somebody, we hire two people at
half time. We don't hire one person. And the reason for that is so that there's redundancy,
so that there's backup. So if somebody leaves, somebody else can come in. But also, so two people can
talk to each other about the project. And it's not somebody just working on this in isolation.
So that's a kind of a conscious choice that we've made over time.
Okay, so next to last slide is just an archite of where we are. So there is a few people that are
kind of up at the top leading. We've got a bunch of different groups that are focused in different areas.
And yeah, and I think things are generally going well. And we are able to build different skills
and different areas that help with what our collaborators are interested in.
So to wrap up, we've had a bunch of successes. I think at this point,
this software director is about a fifth of the organization in terms of both staff and funding.
And I think that's probably a reasonably healthy, healthy size.
We haven't had anybody that had to leave because of lack of funding in at least 10 years.
And I'm not sure if anything happened before that because that's the time that I've been there.
I mean, obviously people have left for a variety of reasons, but nobody has had to leave because we
didn't have the funding for them. And I think that's quite important in telling people that this
really is an institution that they're joining and not a three-year project they're joining for example.
We have, I think, quite a happy staff and a good culture and people see opportunities for
personal and technical growth. And I think we're making an impact on research at Illinois in the
country and on the RSE movement. We do have a couple of challenges though.
So one of them is funding. The current external environment is a quite a large challenge.
I don't know if you've heard, but there's some stuff going on in the US, and it's made this a
little bit difficult. So we're not exactly sure for a variety of reasons, how much funding
we're going to be getting in from different government sources and philanthropic sources and
industry as well. And so there's lots of different effects that are leading to challenges.
We also have almost no institutional support from the University as a whole in terms of funding.
And there's other RSE groups that are very heavily supported by their institutions.
So we're supported by projects. We just have a lot of projects and we can kind of build up
critical mass enough that this works. But if it stops working, then we don't have the fall back
of the institution. So we'll see what happens with that. And then the other thing is that hiring
and AI are both making our existing processes difficult. So hiring is becoming more challenging recently
and AI is making everything more difficult. I guess is a short way of saying it as we've heard
in previous talks. So just as an example, we used to have a pre-screening for hiring or we'd
ask people to do a programming exercise, which doesn't make any sense anymore because they just
ask an AI agent what the answer is. And so we don't know how to evaluate people in a good way
until we actually know them, until we meet them and talk with them. And AI as a whole also leads to
questions about with our collaborators. If they think that they can just use AI to write software
themselves and they're going to write good software that's sustainable, then do they need us?
And how do we convince them that they actually still do? And that there's something that we're
doing that's different. So we do have some challenges, but that's where all these things
and happy to answer any questions. Thank you.
A couple of minutes? Yeah, go ahead.
Yeah. So the question is kind of basically, where do RSEs fit with what we're going to do?
Within the organization and there's kind of pros and cons in different aspects. And so I think
there's a few different models as you mentioned. In some cases they report to the IT organization.
In some cases they report to the research organization when there is a central research
organization, which is the case at Illinois. At some cases they're more related to computer
sciences. It seems that there is not an answer for this at all and it really depends almost entirely
on the institution. And even when it happens in an institution it's not necessarily the thing
that you might say from the outside is best, but for some reason the people in the institution
have thought it's the way that it's going to work. So there is a lot of discussion about this.
And I think it's another thing where there's a small number of models and they have pros and cons,
but it almost more depends locally on what makes sense rather than these kind of global
questions. One more question? Yeah. Go ahead. Sorry.
Down. Sorry. Yeah. Sorry. I saw you first. Yeah.
But hiring would record open source contributions in a way. Yes. And follow up. Yeah.
Because open source was mentioned when they took out of it. Yeah. Yeah.
A property of your project. Yeah. Are it open? Yeah. Okay. So let's see. So is open source
work a something that kind of can be brought in in the hiring process and can be
evidence of somebody's qualifications and answers completely yes. And ideally that happens and
we're very happy when that happens. But sometimes we have people that are coming from industry
and they can't actually share any code or or something else. So it's we like to do that when we
can, but it isn't always an answer. And then in the comment that's completely true that I didn't
really mention open source, which is kind of a strange thing here. But I will say that from
my point of view, I think there's research software kind of as one space and there's open source
is another space and they overlap a lot but not completely. And I think we would fall into that in
which probably 90% I would say of what we do as open source. Not everything. There's some research
software that we work on that's not, but most of it is and that's what we try to do. And once
somebody is add and see I say that is something that we see as something to strive for is
is working openly and when we have projects that are open we try to start them as open from
the very beginning. And so we try to follow the best open source practices we can with the
recognition again that some projects that's not the right answer for even though almost all the
time it is. There's a couple where it's not. Did you want to? Yes. What about, I used to
let me know what it, the RSC doesn't have control over the direction that they have any and what
about do? Yeah. So question about the RSC's and control over their own research. So this is actually
a tricky question because we have another track that besides the RSC track that's called a research
and those are people that do have complete control over their own research but have a much
different model and they get evaluated based on papers more often rather than software although it
can be both where the RSCs are getting evaluated almost entirely on software. We don't have a
great answer for people that are in the middle other than as people get to be more senior and they
can bring in their own projects or create their own projects then that they have the flexibility
to do what they want with those projects. So in some ways if you can bring in funding to support
something then you can do that as long as it doesn't seem like it's tremendously out of line with
everything else as long as it lines up with the institutions objectives. So if somebody wants to do
some research that's related to software and they can get funding for it they will have time to do that
but it's hard to start that up because we don't really have kind of seed funding or a way to
do that so it almost has to to come out of a collaboration with somebody else or it has to be
something that's such a great idea that you can write a proposal without having done much work
and it happens but it typically is people that are more senior that have had some experiences
that they can build on and turn that into a proposal for an idea. Okay one more I'm not sure where we are
in time. Okay so I go ahead sorry. I have the feeling that I've been working a lot with research
and I'm not a good collaboration but somehow the open-minded community is not a very good
extended from this network. So sometimes occasionally the people become plastic if you can actually
include them within the process. So because I have a feeling that you cannot force or resist them
somehow or at least introduce some changes. We didn't have the software either for one minute
because it's either the software community better so come to community so we can meet here
the data and the feedback also to the community. Yeah so the question is basically kind of how
there's an appearance that we're not interacting with the larger open source community
and the open let's say open data community. I'm not sure if that's exactly fair but let's let me try
of that one for a minute. So we I think we do interact with them but we don't necessarily
talk about it very much. We kind of do it more as individuals as opposed to as an organization.
So a lot of the people that are working on these projects are using other open source projects as
part of them and they're contributing back to those often as individual contributors and in that
the time that they're using to do that is time that the project effectively is paying them
for. It's not kind of volunteer effort that they're doing on the side. So I feel like we do
participate in a number of projects in some cases some of the things that we do we try to
turn them into larger, more general open source projects. We're often not as successful with
that as we might like and that could be from lack of experience. It could be because things that
we think are general are still somewhat specialized. I don't have a good answer but I would say
that we do try to do this but whether or not we're as successful as we could be probably as a
question. I appreciate the comments and I think as individuals we do this well as an organization
we probably don't prioritize it very much. We don't talk about it at least as enough.
Okay, thank you.