Synthesis Infrastructures - User contributions [en]

Interdisciplinary Models

2022-11-13T15:36:40Z

EllieDeSota:

{{Group
|Decription=How do we define minimal information models tuned for synthesis that can interoperate across various disciplines?
|Topics=Interoperability
|Discord Channel Name=#interdisciplinary-models
|Discord Channel URL=https://discord.com/channels/1029514961782849607/1040385502026682408
|Facilitator=Wayne Lutters
|Members=Elianna DeSota, James Howison, Konrad Hinsen, Leo Ware, Paul Itoi, Peter Murray-Rust, Wayne Lutters
}}
== What ==

How do we define minimal information models tuned for synthesis that can interoperate across various disciplines?

Concrete problem expressed by [[Peter-Murray Rust]] here: https://discord.com/channels/1029514961782849607/1040214388554084372/1040299259930611833: "The idea of Hypothesis testing is common in some disciplines, unknown in others. For example chemical synthesis or materials science is "can we make X?" and many sciences are exploratory - what can we see with a new telescope, plants in Antarctica, etc. You have to design your project but I suspect Hypothesis doesn't come into it."

And to a certain extent, the issue of representing/discussing the discourse of computational research (e.g., model parameters), discussed by [[Konrad Hinsen]] here: https://discord.com/channels/1029514961782849607/1038988750677606432/1039576903838859326

This connects also with [[Peter-Murray Rust]]'s work on [[Semantic Climate]] (semantifying the IPCC report).

* see discussion here: https://discord.com/channels/1029514961782849607/1040057721044598788/1040060670907002973
* and here: https://discord.com/channels/1029514961782849607/1033091746139230238/1040226423346040853

And also connects to emerging discussions around interoperability and Surfacing/managing/resolving disagreements in ontologies/terms/federation

=== Initial discussion ===

Matthew, Peter, Wayne, James, Ellie, Leo

Projects discussed:

- Scraping literature in geosciences to spatially map out relevant variables and contributions across disciplines http://globe.umbc.edu/
- Materials Genome Initiative mentioned: infrastructure well-supported but still siloed
- OPTIMADE: common API format between existing materials databases

"grassroot tech assemblage can work at scale. "Shoddy" now works." Shoestring budgets driving open source innovation.

"need to flourish long enough to been seen by other disciplines" Idea that each of these initiatives have a typical academic funding life of 3-7 years and then are sunset. Do they twinkle in the sky long enough to be seen by other disciplines? Core sustainability issues not just of the tools / platforms but of the motivating ideas beyond them.

"So: how do we work in a way that others can learn from in future" -- without being discouraged from starting new things, encourage high-risk, high-reward innovations.

Reaching plateau of open data --- metrics on who is using and what using for

Similar challenges in enterprise: what data do we have within an org, and who is using it? https://data.world/ vs more public initiatives like https://coleridgeinitiative.org/

Insight around longevity -- is it the infrastructure that lives on? the vision? or the relationships? Unique value of EC framework initiatives (e.g., Horizon 2020) that are as much political projects as they are scientific ones. Those connections between people and labs persists.

Discovering and forming communities of practice around datasets -- how does one person's use leave traces that others can discover? How do we align the challenges across time (ie my experience when I was grappling with a specific column in a dataset, aligned with someone doing just that a year later).

Academic model of competition rubs against open science --- both in sunk time and possessiveness of data

Disciplines have different reductionist traditions, what is the well-defined focus of study. Is this the substrate that enables cross disciplinary data engagement?

Where do people gather to have these conversations? What are the communities of practice, publication venues to share knowledge about working across the disciplines? Where do these happen within disciplines and where is the meta-science narrative developing? Historically in e-science-like funding; especially international laboratories

Open notebook science https://en.wikipedia.org/wiki/Open-notebook_science<nowiki/>-- show the world as you are doing it, make connections on the day of publication. Very well defined strategy with templates. http://opensourcemalaria.org/

Bold vision of what is possible, e.g. automated recombination & discovery: https://materialsproject.org, https://materialsproject.github.io/fireworks/ - would also like to plug OPTIMADE here, which is then unifying datasets between several endeavors in this field

Domain differences between contributing individual data points vs entire datasets

Can grassroots emulate giant centralisation within industrial monoliths

Analogy between web frameworks/OSS: emerging from many hands working towards similar problems

Gift economy of software applied to data? Frictionless data as an example https://frictionlessdata.io/

Collectivization as a model -- being able to push upstream to graphs at different scales

Incentivizing collectivization

'''Ellie + Leo thoughts during break -'''

Seems like there is a three fold problem,

- easy to share data/info

- easy to use data/info

- easy to cite data/info

Right now - none of this is free. It takes so much time to actually find all the weight of evidence, and to connect all the data. And also,.. the finantialization model which attempts to make this 'incentivized' at a greater scale seems relatively meh? Unless the returns are pretty big, I feel like I am much more likely to be lazy than to care about a few extra ETH. In addition, just thinking about reputation networks feels like it should/would ened to be more connected to your actual community in which you had established systems of practice.

Infrastructure -> Make an overleaf that automatically brings in citations? How do we sync GPT into this?

What is an equivalent of GPT for data? Where you want to look for specific DATA - you aren't concerned initially with the original questions asked to the pieces of data.

SUPER cool project we should talk about -> DeSci Labs.

'''Identifying key questions:'''

- Which solutions have worked in other domains?

- What are the differences between (ontological, socio-political, economical) domains that lend themselves to different solutions?

- Extending the concept of "discipline" to e.g., cataloguing human infrastructure (cities, roads etc), "Discipline as a search across a reasonably well defined search space". Possible with a small number of disciplines (e.g. medicinal plant chemistry needs three domains - all with good ontologies)

- Alignment of primitives --- example of plants in expressed different locales and the effect on local climate

- Aligning communities of practice with a wider goal?

'''Possible outcomes of this group'''

- Compendium of practices in different fields

--- Collection of venues: where are the discussions happening now at the discipling and meta level

--- Collection of case studies around primitives in different disciplines

- Collecting ideas from other attendees from disciplines within the workshop in a survey: how would you/your field do things differently were all these things in place? Different life cycles and capturing nascent knowledge

'''Key themes for reporting back:'''

- Longevity and sustainability

- Lowering initial costs

- Designing work such that it can contribute upstream to a "practice"

- Differences in solutions by scientific disciplines, mechanisms of production, budgets, motivations and governance

- Alignment of primitives within a discipline: do you contribute a data point or a dataset? Different approaches required

- Synergies with other groups: interfaces, graphs, social systems

- Didn't really mention papers once!

>>> DAY #2 RUNNING

1.) STRUCTURE & USE
IPCC -> claims obsidian
beyond info artecure
holy scripture
find new connections
index
venues
women for open climate
NAS reports

oil exploration & production
well logs

2.) MOTIVATION
indisciplinary doing primary science together
infrastructure vs. primary science
front line science vs. backstage
FAIR, need joint infrastructure
pushing upstream
data accessibility
desci / halogen
community structure, customizable, constrained
paper replication

synthesis study <<< STILL CAN STUDY THIS
karen baker not just PIs, get right level on team
edge researchers do the actual work core business doesn't care about
funding on topics PIS don't care about
MIT look-it, psychology

3.) INFRASTRUCTURE
research external to academica
crowdsourced science, what happend to that?
Infrastrucutre
research librarian / career path / dev
infrastructure -- machine shop, glassblowers, super computer centers
Pooled resources
The SW people
eSci UK - roles $ sw sustainability institutes

"when infrastructure becomes a research project itself vs. a resource for community often drifts off"
The GRID - Physics
Libraries - pandisciplinarity
Not aware of "embedded librarian" as a thing.
Domain scientist expertise =/= meta data
Platform emergence - core & tipping
Gawer, A., & Cusumano, M. A. (2008). How companies become platform leaders. MIT Sloan Management Review, 49(28).
Too wide a view,
Need one person in each department that understands and can translate that discipline, and something networking them together

4.) MOTIVATION - FUTURE SELVES
Motivation for FLOSS, reduce the maintenance cost now for anticipated future work
What is it for open science?
Mark up our grant proposals, future reuse
Semantic bibliography
Tremendous value vision, sustainable overhead?
Finding the right structure
Maintaining semantic bibliography for a grant is a great form of legitimate peripheral participation
We plan to try this out for preprints in climate. Sensemaking structure to 15,000 refs

=== DAY 2 ===
question how can we use AI processing to discover non-explicit connections?

- claim - the simplest job to get here is to create an index.

question what are the venues where we can use open/interdisciplinary science?

- evidence - women for open climate already exists and would would support this

claim - we conceptualize inter-disciplinarity as multiple groups doing their primary research together.

- evidence - (counter to truth of claim) Django as an example. This is collaboration btwn website builders making the infrastructure. Or scikitlearn etc. None of these are collaboration on their primary activity.

- question - how might we ensure that these groups can collaborate on projects that aren't their direct work?

claim - there is a disconnect between the team level incentives and the global level need for collaborative infrastructure

- evidence synthesis centers couldn't resolve this algorithmically, and so simply needed to create a space where everyone came in to resolve the connections.

Centralized pool for resources - the university used to have machine labs or computing centers meaning that the university cuts it back bc its too costly

- Documenting these stories.

claim - Teaching mission of the university is missing, I think they should engage in teaching during setting up these resources.

- evidence - super computing centers used to do this, but then get's lost.

Research Software Engineering groups -

- They get attached to grants and become the software engineering groups.

- evidence - the escience program in UK moved into software sustainability institute.

- - There was an atmosphere that this was super interesting vs a dedicated national service.

- evidence - the grid - came out of particle physics. Everyone needs to buy into it. It's been largely overtaken by major software companies.

claim - libraries have attempted to make sure that you can manage all the different data but scientists need to know super deeply what the data is in itself, and a library can't actually provide this service. You NEED domain specialists. Similarly, there can't just be a 'random' person who is focused on integrating data -> they won't be able to give you this info.

- evidence - how platforms emerge. Coring + tipping. The breadth of view is so large that they aren't going to spot the core of the data.

- evidence - cambridge at the institutional level can enforce data sharing. BUT sharing it isn't necessarily what makes it useful.

- Claim - we're not making the most of the networks that we have right now, it feels like you need a single person who has the knowledge to share and integrate data to the 'institutional' repository and then the institutional repository can be integrated at the more global level.

- is there not enough interest for this to happen?

- claim - main motivation for open source is to reduce the maintenance cost of things we build on in the future.

- evidence - when you write a grant application, you write a lit review but then when you need to write an updated lit review -> you need to bring a 1.5 year out of date lit review that you haven't explored.

OPEN QUESTIONS -

- How do we identify the goldilocks level - wide enough infrastructure to allow for everyone to be on the infrastructure, and narrow enough for specific groups to be able to customize to their ideal needs.

- evidence - data analysis, genome searching, these are places where everyone from many disciplines come together to ensure that you have a shared tech which brings ppl together.

Talk:Interdisciplinary Models

2022-11-13T14:48:12Z

EllieDeSota: Single shot incentive structure understanding?

- ![](https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2Felianna%2FJBFV34v82l.44.12%20AM.png?alt=media&token=eb439538-732e-4b74-8506-efac18ed219a)
- A diagram that represents broadly the way I think about incentives at three relevant levels for scientific collaboration. #open are these the only levels of scientific collaboration?
- Through this diagram, I argue that incentives are rather inherently not scaleable, and different incentives are more in 'action' during different levels of collaboration than they are at other levels.
- For example,** at the level of the team**, one way to explore the explanatory challenge of why someone might spend huge amounts of effort doing something that they aren't paid for such as peer review is that among their community - they've cultured an ethic of contributing to the more abstract 'global scientific community' (cite). Or alternatively - the delivery on creating high quality scientific practices which aren't incentivized on a global level is more likely to grow from a community like the one here, one that has created a mission in developing better practices __despite__ the incentives that don't support these practices.
- On a slightly higher level -** at the level of the institution **- incentives revolve at the moment around securing tenure, guarenteeing secure funding for your projects and ensuring that you can continue on your work and remain supported. These mechanisms can be direct through social incentives to adhere to various practices in order to be seen as a valuable collaborator (if the institutional community has established practices which you can do to 'navigate' relationships external to your main team) or through monetary/survival incentives which require to to navigate communication, bureaucracy, and regulation to ensure you remain part of the institutional structure.
- I think of Universities like this. There can be university ethics around Open Science and creating a practice of high quality science, and there are also the general rules that are requried for a professor or academic to be apart of the university. Essentially, the university as a institution gets to decide if they grant you tenure, a salary, or a lab in which to work. If you haven't adhered to the rules they have set up (either through direct action, or through no fault of your own) than as the group with power they can use that power to exclude you the academic from the institution + and resources.
- At** the highest level**, these incentives are even further extracted. The most social of these incentives can be seen as 'memes'. Open Science right now seems to be a meme - almost everyone I talk to thinks about how science isn't achieving it's goals, but given the strength of the institutional and team level communities, unless an individual is quite taken by the importance of the meme and is willing to be the person to start that social norm within their team/lab/university they are more likely to adhere to the more direct social rules and regulations which help them navigate their more immediate community. On this level you can set up infrastructures that allow for wider scientific communication and make it easy/standardized/and integrated into scientific practices to practice interdisciplinary best practices, but you likely can't force adoption. In addition, you can create practices of assigning higher clout/reputation/payoffs to those who adhere to the global social memes and create changes within their process, but if a smaller community doesn't require it or actively discourages it (I'm thinking Pharma, few Pharma scientists at this point will be part of the open science movement because their institutional incentives (having a job, lab, etc) are often dependent on their zipped lips).
- The patterns I currently am averse to at this level is the initial adoption of such structures by the most capable. #open Is this bad??. We see this in open source - those with the most time and priviledge are currently the only ones contributing bc those who are alreadly systematically disadvantaged are more likely to get into positions where they have to compromise the moral of sharing information in order to secure a more mainstream or secure job. My hypothesis for why I find this distasteful is because each new economy is established almost entirely by those who were in power in the previous economy (CITE - via Andre's book/convo). This means that the likelihood that these new economies replicate the past economies seems super high. I find this particularly true in the current case of the gig-economy where many people were like - THIS IS GREAT! give everyone the flexibility they need, but this actually created a space where people weren't supported and were spending all of their time trying to find the gigs, or survive on the marginal returns. Thus the powerful - had their security and got some extra, but those who needed the support were now even more a slave to a system which didn't sufficiently compensate them. (CITE??)
- Understanding these levels is essential for understanding how I think about the failings of interdisciplinary science at the moment. Which has more levels - of course :).
- When thinking about sharing among different people there are three potential failing points.
- 1. We don't share our knowledge
- 2. We don't use the shared knowledge
- 3. Our shared knowledge isn't exhaustive of the perspectives that need to be shared
- For the first, Open Science right now seems to go some way towards finding ways to ensure that we share all of our knowledge. Pre-print servers, changing scientific practices arising in a rapidly publishing scientific landscape headed up by genuinely massive journals such as PLOS () or Elife (). But as far as I can tell is that it fails to allow us to actually use or check the exhaustiveness/inclusiveness of this knowledge.
- For the second point - there are many more breakdowns. https://www.arthurperret.fr/articles/2022-11-13-researchers-needs-and-options-for-collaborative-synthesis.html - this is a LOVELY article that expresses what's important here but I'll also extract out.
- We need primitives that can be expressed across disciplines. This means FAIR data principles essentially.
- We need to be able to Find, Access, Interoperate (connect), and Reproduce data insights. #open - which of these are least explored?? My hypothesis -> find + interoperate are most difficult. I see reproduce + access solved through the open science stuff and ESPECIALLY through the DeSci Labs work.
- Data that can be aggregated across all of science
- Now I'll bring in the incentive levels from above. The act of integration happens at the individual level. The act of using integrated knowledge happens at the global level. #open is this actually a problem?.
- My current assumptions is that this interdiscipinary synthesis is really difficult because if at the small level you don't have a system to integrate into the global system, than the data won't be interoperable. But also, if you are a small group - you will almost INEVITABLY have a contextual situation in which youre systems are not necessarily compatible with a more 'general' audience. #open - how has this been resolved in the past??

Interdisciplinary Models

2022-11-12T17:17:06Z

EllieDeSota: Random thoughts that we were thinking about during the break

Interdisciplinary Models

2022-11-12T17:09:40Z

EllieDeSota: Additional things

Abstract Poetry

2022-10-31T13:44:58Z

EllieDeSota: Added some documentation to make the project understandable

<nowiki>**</nowiki>Hey hey! The main idea of Abstract Poetry is down below - it's a bit more pitch focused because we wrote it for an application, but hopefully it helps make what we've done so far clear and some of the limitations so anyone can know how/if this can be useful for whatever we end up collaborating on. Cheers!{{Project
|Homepage=abstract-poetry.fly.dev
|Description=What if search didn’t stop at keywords? Abstract Poetry hopes to facilitate exhaustive search without relying on matching exact keywords to papers. By focusing on a paper by paper search process that learns the types of paper's you're most interested in, we hope to eliminate the need for biasing algorithms and create a faster search process with a less overwhelming search interface. Right now - we want to make this useful! Could we integrate it with IPFS to begin a semantic decentralized search platform? What features might make this a tool that makes researcher's lives easier? How might the interface help aggregate and create more robust and checkable links to evidence? If you have thoughts let us know!
|Repository URL=https://github.com/curl-projects/abstract-poetry
}}

== What’s our Main idea ==
Academic search is currently a giant spreadsheet which associates every academic paper with a set of keywords and metadata and waits for papers to be called upon by researchers.

This data structure asks researchers to build a mental model of all the relevant research in their sub-field and associate that research with the key metadata required to access it. Holding lots of arbitrary information in our heads is something that we as humans aren’t very good at, so it takes years in one field for researchers to consistently do this well.

To make this process easier, existing search platforms try to bring the most important results to the top of your search results. But computers aren’t very good at this. Most metrics of importance are actually metrics of popularity, which systematically biases search towards ‘hot topics’ and well-marketed research.

Abstract Poetry flips this distribution of work. We ask humans to do what they’re good at: setting criteria for search determining what’s important to them. And we ask computers to do what they’re good at: storing, processing and filtering millions of data points based on those criteria.

We’ve done this in two ways:

* '''We’ve rethought the interface for search''' In our search, academics tell us how relevant each returned result is which dynamically improves their future results. To represent their exploration, we return an interactive visualization of the connection between the researcher’s preferred results and their disliked ones. This gives them the opportunity situate their preferred research within the context of all the results, relevant and irrelevant, within the domain.

* '''We’ve given the computer a way to understand and categorize a continuous map of science.''' We’ve partnered with Semantic Scholar, who has given us access to a semantic embedding database which holds 768-dimensional semantic embeddings for 140 million papers. Collectively, these vectors create a high-dimensional map of science that represents scientific domains in terms of how similar papers are to each other. We explore this map of science using a Bayesian multi-armed bandit algorithm called Thompson sampling, which uses the researcher preferences (”More Like This/Less Like This”) to identify regions of the embeddings space that the researcher is currently interested in.

In the last two months, we have taken Abstract Poetry from an idea to two workable products (a general search and an interactive bibliography), completed 40 user discovery and product test interviews , and developed a set of internal systems which help us turn user problems into updates in our product.

What we’ve done so far is promising, but limited.

# '''PLOS is only 0.1% of science.''' This limits our current search and prevents many researchers from being able to get quality results from our tool.
# '''We are limited to Semantic Scholar’s semantic embeddings.''' From user interviews, we’ve learned that knowing contrasting papers and the types of similarity between papers such as methodology, claims, or topic would be essential to improving the search experience. Semantic Scholar’s embeddings are not trained to provide this level of detail.
# '''We have not documented our model.''' Several of us work multiple jobs, meaning that the essential work of documenting the work we’ve done has been pushed to the side.

Abstract Poetry

2022-10-31T13:30:06Z

EllieDeSota: Added some info about the project.

{{Project
|Homepage=abstract-poetry.fly.dev
|Description=What if search didn’t stop at keywords? Abstract Poetry hopes to facilitate exhaustive search without relying on matching exact keywords to papers. By focusing on a paper by paper search process that learns the types of paper's you're most interested in, we hope to eliminate the need for biasing algorithms and create a faster search process with a less overwhelming search interface. Right now - we want to make this useful! Could we integrate it with IPFS to begin a semantic decentralized search platform? What features might make this a tool that makes researcher's lives easier? How might the interface help aggregate and create more robust and checkable links to evidence? If you have thoughts let us know!
|Repository URL=https://github.com/curl-projects/abstract-poetry
}}