Computable Graphs - Revision history

Joelchan: /* question What additional context beyond provenance could be useful? */

2022-11-18T14:18:29Z

question What additional context beyond provenance could be useful?

← Older revision		Revision as of 14:18, 18 November 2022
Line 130:		Line 130:
	DeFacto tool for deep fact validation on the web: [[/github.com/DeFacto/DeFacto\|https://github.com/DeFacto/DeFacto]]		DeFacto tool for deep fact validation on the web: [[/github.com/DeFacto/DeFacto\|https://github.com/DeFacto/DeFacto]]

	==== [[question]] What additional context beyond provenance could be useful? ====		==== [[question]] What additional context beyond provenance could be useful for scholarly knowledge graphs specifically? ====

	* Concept drift over time (in general entity/relation evolution over time)		* Concept drift over time (in general entity/relation evolution over time)
	* Seems like discourse links across papers don’t neatly fit the provenance definition, but are an aspect we want to capture?		* Seems like discourse links across papers don’t neatly fit the provenance definition, but are an aspect we want to capture?
	* Are there other aspects that provenance doesn’t cover?		* Are there other aspects that provenance doesn’t cover?
			* Some key pointers in Bowker (2000): https://joelchan.me/INST801-FA22%20Readings/Bowker_2000_Biodiversity%20Datadiversity.pdf

	==== Links to NLP research ====		==== Links to NLP research ====

Joelchan: notes from day 2

2022-11-13T17:33:47Z

notes from day 2

← Older revision		Revision as of 17:33, 13 November 2022
Line 130:		Line 130:
	DeFacto tool for deep fact validation on the web: [[/github.com/DeFacto/DeFacto\|https://github.com/DeFacto/DeFacto]]		DeFacto tool for deep fact validation on the web: [[/github.com/DeFacto/DeFacto\|https://github.com/DeFacto/DeFacto]]

	==== What additional context beyond provenance could be useful? ====		==== [[question]] What additional context beyond provenance could be useful? ====

	* Concept drift over time (in general entity/relation evolution over time)		* Concept drift over time (in general entity/relation evolution over time)
Line 137:		Line 137:

	==== Links to NLP research ====		==== Links to NLP research ====
	~~Biggest open questions seem to be: scalability vs expressivity, multiple modalities, updation~~		Seems like there's a bit of a gap between NLP research

	~~Scoring functions in KRL: In representation learning for KGs, models~~ are trained to embed entities and relations in a common space and then score (subject, predicate, object) triples according to likelihood - can these scores be use as additional “plausibility” or “validity” metadata from a provenance perspective? However, these scores are usually produced by models that are learning to replicate the ~~structure~~ of ~~the~~ knowledge graph~~, so they can’t link back to any textual evidence to support the judgements.~~		[[question]] What are the points of overlap between subproblems around scholarly knowledge graph provenance and existing NLP research?

	~~Auxiliary knowledge in graphs: [https://ieeexplore~~.~~ieee~~.~~org/stamp/stamp.jsp?tp=&arnumber=9416312 <nowiki>~~[2]~~</nowiki>~~] ~~lists some work that stores textual descriptions alongside entities, entity images (image-embodied IKRL) and uncertainty score information (ProBase, NELL and ConceptNet).~~		(probably a rich opportunity for a position paper - could bei nteresting to NLP venues; e.g., [[NeuRIPS AI for science workshop]]: possible contact: Kexin Huang

	Temporal knowledge graphs: [2] is a survey of knowledge graphs from an NLP perspective and highlights temporal knowledge graphs as a key area of research. Temporality is one kind of provenance information. Current directions being explored include		* Biggest open questions seem to be:
			** scalability vs expressivity, multiple modalities, updation
			** Interpretability or transparency: Provenance information can be used to make KGs more transparent, because we have sources that provide evidence for why this fact/relation holds true.
			** Commonsense angle: Another broader application where storing context might be helpful could be for commonsense knowledge graphs?
			*** Might have an analog in terms of "scholarly" (un)common sense, esp. across disciplinary boundaries (but this also has analogs in cross-cultural barriers to communication)
			* Maps to things with some existing body of work
			** Scoring functions in KRL: In representation learning for KGs, models are trained to embed entities and relations in a common space and then score (subject, predicate, object) triples according to likelihood - can these scores be use as additional “plausibility” or “validity” metadata from a provenance perspective? However, these scores are usually produced by models that are learning to replicate the structure of the knowledge graph, so they can’t link back to any textual evidence to support the judgements.
			** Auxiliary knowledge in graphs: [https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9416312 <nowiki>[2]</nowiki>] lists some work that stores textual descriptions alongside entities, entity images (image-embodied IKRL) and uncertainty score information (ProBase, NELL and ConceptNet).
			** Temporal knowledge graphs: [2] is a survey of knowledge graphs from an NLP perspective and highlights temporal knowledge graphs as a key area of research. Temporality is one kind of provenance information. Current directions being explored include
			*** Temporal embeddings: extending existing KRL methods for quadruples instead of triples, where time is the fourth entity
			*** Entity dynamics: changes in entity states over time
			*** Temporal relational dependence (e.g., a person must be born before dying)
			*** Temporal logic reasoning: graphs in which temporal information is uncertain.
			** Knowledge acquisition: Some work has tried to explore jointly extracting entities/relations from unstructured text and performing knowledge graph completion. This can be a useful way to jointly model unstructured text from previous papers that do not have associated graphs and papers for which discourse graphs have been annotated. In general, joint graph+text embedding techniques seem useful. Additionally, a technique often used to train models to extract relations from text, distant supervision, can probably be leveraged to detect and add provenance information post-hoc. The idea behind distant supervision is to scan text and find sentences that mention both entities involved in the relation, considering such sentences to be potential evidence sentences. But this is a noisy heuristic - a sentence mentioning two connected entities may not be mentioning the relation between them - and will likely require some curation.

	* Temporal embeddings: extending existing KRL methods for quadruples instead of triples, where time is the fourth entity
	* Entity dynamics: changes in entity states over time
	* Temporal relational dependence (e.g., a person must be born before dying)
	* Temporal logic reasoning: graphs in which temporal information is uncertain.

	Knowledge acquisition: Some work has tried to explore jointly extracting entities/relations from unstructured text and performing knowledge graph completion. This can be a useful way to jointly model unstructured text from previous papers that do not have associated graphs and papers for which discourse graphs have been annotated. In general, joint graph+text embedding techniques seem useful. Additionally, a technique often used to train models to extract relations from text, distant supervision, can probably be leveraged to detect and add provenance information post-hoc. The idea behind distant supervision is to scan text and find sentences that mention both entities involved in the relation, considering such sentences to be potential evidence sentences. But this is a noisy heuristic - a sentence mentioning two connected entities may not be mentioning the relation between them - and will likely require some curation.

	~~Interpretability or transparency: Provenance information can be used to make KGs more transparent, because we have sources that provide evidence for why this fact/relation holds true.~~

	~~Commonsense angle: Another broader application where storing context might be helpful could be for commonsense knowledge graphs?~~

	[[/link.springer.com/chapter/10.1007/978-3-642-13818-8 32\|Application to finding evidence for clinical KBs]]		[[/link.springer.com/chapter/10.1007/978-3-642-13818-8 32\|Application to finding evidence for clinical KBs]]
Line 176:		Line 180:
	Interesting question around score for contributions to the scientific KG instead of document-centric metrics like h-indices (potentially more accurate assessment of contributions)		Interesting question around score for contributions to the scientific KG instead of document-centric metrics like h-indices (potentially more accurate assessment of contributions)

	[https://www.researchgate.net/profile/Soeren-Auer/publication/330751750_Open_Research_Knowledge_Graph_Towards_Machine_Actionability_in_Scholarly_Communication/links/5c5e9bc7a6fdccb608b28f6f/Open-Research-Knowledge-Graph-Towards-Machine-Actionability-in-Scholarly-Communication.pdf <nowiki>[4]</nowiki>] has a really nice analogy: when we moved from phone books and maps were digitally transformed, we didn’t just created PDF books with this information, instead we developed new means to organize and access information. Why not push for such a re-organization in scholarly communication too? Building on [3], their ontology also contains: problem statement, approach, evaluation and conclusions. They use RDF with a minor difference: everything can be modeled as an entity with a unique ID. Limited subset of OWL (subclass inference) is supported. Small user study with this tool, but it doesn’t seem like Dafna’s question about utility of KG constructed by someone else or consensus across people were measured.		[https://www.researchgate.net/profile/Soeren-Auer/publication/330751750_Open_Research_Knowledge_Graph_Towards_Machine_Actionability_in_Scholarly_Communication/links/5c5e9bc7a6fdccb608b28f6f/Open-Research-Knowledge-Graph-Towards-Machine-Actionability-in-Scholarly-Communication.pdf <nowiki>[4]</nowiki>] has a really nice analogy: when we moved from phone books and maps were digitally transformed, we didn’t just created PDF books with this information, instead we developed new means to organize and access information. Why not push for such a re-organization in scholarly communication too? Building on [3], their ontology also contains: problem statement, approach, evaluation and conclusions. They use RDF with a minor difference: everything can be modeled as an entity with a unique ID. Limited subset of OWL (subclass inference) is supported. Small user study with this tool, but it doesn’t seem like Dafna’s question about utility of KG constructed by someone else or consensus across people were measured. But they do speak to Dafna's question about how painful it is to construct KGs.

			so possible good open [[question]]: how might KGs constructed by one set of scientists actually be useful or not for other scientists, and what factors influence whether this is the case?

			* [[experiment]] idea: even just starting by compiling any successful instances of adoption of KGs taht span multiple groups would be a good thing to do; [[Peter Murray-Rust]]. maybe [https://www.nlm.nih.gov/research/umls/index.html UMLS] also

	Note: Both papers actually reference a fair number of other works in this direction, which could be good follow-up reading		Note: Both papers actually reference a fair number of other works in this direction, which could be good follow-up reading

	=== Papers I didn’t get to ===		=== Papers I didn’t get to ===
			(numbering here does not map to numbering in sections above)

	[1] [https://pdfs.semanticscholar.org/713b/b398b85b034f2139e08b4ca0f7791fd545bc.pdf?_ga=2.19474043.2123037593.1668226180-648530244.1660769576 Empirical study of RDF reification]		[1] [https://pdfs.semanticscholar.org/713b/b398b85b034f2139e08b4ca0f7791fd545bc.pdf?_ga=2.19474043.2123037593.1668226180-648530244.1660769576 Empirical study of RDF reification]

Line 206:		Line 216:
	* reuse of records/data/stuff in general in CSCW across a bunch of work domains/settings, like aircraft safety, healthcare, etc. (generally talks about why it's hard to reuse stuff from others, need context)		* reuse of records/data/stuff in general in CSCW across a bunch of work domains/settings, like aircraft safety, healthcare, etc. (generally talks about why it's hard to reuse stuff from others, need context)
	* annotations / social reading (generally negative/mixed)		* annotations / social reading (generally negative/mixed)
			MG: can we search the literature based on what tool they studied things with?

			NO. not right now.

			lack of good tools to "refactor" the literature to match those queries

			maybe like Elicit? https://elicit.org/search?q=What+is+the+impact+of+creatine+on+cognition%3F&token=01GHRSBKG6P1R9343NBP2ZWZ5B&c-ps=intervention&c-ps=outcome-measured&c-s=What+tool+did+they+study but for everything??

	== Compiling graphs to manuscripts ==		== Compiling graphs to manuscripts ==
Line 245:		Line 262:
	* the majority of empirical research papers in biology have a similar structure (question/ motivation/ evidence (fig.1a)/ claim for each paragraph & figure panel)		* the majority of empirical research papers in biology have a similar structure (question/ motivation/ evidence (fig.1a)/ claim for each paragraph & figure panel)
	* multiple researchers (or students) asked to highlight the questions/claims/evidence text from a paper will highlight similar/consensus text (part of the NLP-to-highlights project)		* multiple researchers (or students) asked to highlight the questions/claims/evidence text from a paper will highlight similar/consensus text (part of the NLP-to-highlights project)
			notes/questions

			* may need more than just the sentences, but also mapping to intra-project/paper elements, like figures and tables to

			[[File:Screenshot example of GPT-3 task for extracting components from narrative.png\|none\|thumb]]

	== Next Steps ==		== Next Steps ==

Aakanksha: /* Contextualizing knowledge graphs */

2022-11-13T08:42:05Z

Contextualizing knowledge graphs

Show changes

Joelchan: /* Understanding knowledge graph transfer */

2022-11-13T06:14:26Z

Understanding knowledge graph transfer

← Older revision		Revision as of 06:14, 13 November 2022
Line 74:		Line 74:

	== Understanding knowledge graph transfer ==		== Understanding knowledge graph transfer ==
			I was able to track down my main notes/sources from before and clean them up a little (not that much effort :)):

			Last version (not most updated, bc export is slooowwwwww from Roam rn) here: https://publish.obsidian.md/joelchan-notes/discourse-graph/questions/QUE+-+How+can+we+best+bridge+private+vs.+public+knowledge

			Will clean up based on this: https://roamresearch.com/#/app/megacoglab/page/XHvRRSi_0

			TL;DR (can clean up here based on [[Discourse Modeling]] conventions :):

			* distributed sensemaking (Kittur, Goyal, etc.; some promise of yes helpful to share)
			* sensemaking handoff (limited, but a major focus in intelligence analysis, there are tools and processes for doing this)
			* reuse of records/data/stuff in general in CSCW across a bunch of work domains/settings, like aircraft safety, healthcare, etc. (generally talks about why it's hard to reuse stuff from others, need context)
			* annotations / social reading (generally negative/mixed)

	== Compiling graphs to manuscripts ==		== Compiling graphs to manuscripts ==

Joelchan at 19:40, 12 November 2022

2022-11-12T19:40:48Z

← Older revision		Revision as of 19:40, 12 November 2022
Line 5:		Line 5:
	\|Discord Channel Name=#computable-graphs		\|Discord Channel Name=#computable-graphs
	\|Discord Channel URL=https://discord.com/channels/1029514961782849607/1038983137222467604		\|Discord Channel URL=https://discord.com/channels/1029514961782849607/1038983137222467604
	\|Members=Aakanksha Naik, Akila Wijerathna-Yapa~~, Deniz Aydemir, Eran Egmon~~, Joel Chan, Matthew Akamatsu, Michael Gartner, ~~Konrad Hinsen~~		\|Facilitator=Joel Chan
			\|Members=Aakanksha Naik, Akila Wijerathna-Yapa, Joel Chan, Matthew Akamatsu, Michael Gartner, Dafna Shahaf, Belinda Mo
	}}		}}
	Facilitator/Point of Contact: [[Has Facilitator::Joel Chan]]		Facilitator/Point of Contact: [[Has Facilitator::Joel Chan]]

Akamatsm: Flesh out 'compiling graphs to manuscripts' project

2022-11-12T18:27:25Z

Flesh out 'compiling graphs to manuscripts' project

@@ Line 31: / Line 31: @@
 * Belinda: what is scope of the project in terms of users?
 ** Matt: hoping this accessible all the way to high school students also!
-* AI2 has platform for running simulations, make it easy to run and share with others
+* AICS is making platform for running simulations, make it easy to run and share with others - [[Simularium]]
 * Dafna: are there pain points on the input side? (seems painful!)
 * Dafna: how similar are the things that people pull out from the same paper?
@@ Line 59: / Line 59: @@
 * idea of compiling from discourse graph to manuscript seems feasible?
 ** Belinda could mock something up in OpenAI playground really quickly for a few papers
 *** need examples or access to repo
@@ Line 76: / Line 75: @@
 == Compiling graphs to manuscripts ==
 == Next Steps ==

Akamatsm: /* Resources */

2022-11-12T17:52:44Z

Resources

← Older revision		Revision as of 17:52, 12 November 2022
Line 59:		Line 59:

	* idea of compiling from discourse graph to manuscript seems feasible?		* idea of compiling from discourse graph to manuscript seems feasible?
			[[File:Discourse graph to manuscript draft via NLP.png\|900x900px]]
	** Belinda could mock something up in OpenAI playground really quickly for a few papers		** Belinda could mock something up in OpenAI playground really quickly for a few papers
	*** need examples or access to repo		*** need examples or access to repo

Joelchan: /* First breakout group session */

2022-11-12T17:49:19Z

First breakout group session

← Older revision		Revision as of 17:49, 12 November 2022
Line 23:		Line 23:

	=== First breakout group session ===		=== First breakout group session ===
			who was present: Joel, Matt, Michael, Sid, Aakanksha, Belinda
	* anchoring on [[Synthesis center for cell biology]]		* anchoring on [[Synthesis center for cell biology]]
	* challenge: students wrapping their heads around the model; get really excited once they do		* challenge: students wrapping their heads around the model; get really excited once they do
Line 56:		Line 56:
	* idea around changing the reading process somehow (with high hopes for somehting like semantic scholar PD reader that has beginning annotations tuned to what matt is trying to extract, maybe also on the abstract level) --> these could also feed back / forward to other users		* idea around changing the reading process somehow (with high hopes for somehting like semantic scholar PD reader that has beginning annotations tuned to what matt is trying to extract, maybe also on the abstract level) --> these could also feed back / forward to other users
	** can probably start from this: Scim: Intelligent Faceted Highlights for Interactive, Multi-Pass Skimming of Scientific Papers <nowiki>https://arxiv.org/pdf/2205.04561.pdf</nowiki>		** can probably start from this: Scim: Intelligent Faceted Highlights for Interactive, Multi-Pass Skimming of Scientific Papers <nowiki>https://arxiv.org/pdf/2205.04561.pdf</nowiki>
	** cross connections to what [[		** cross connections to what

	* idea of compiling from discourse graph to manuscript seems feasible?		* idea of compiling from discourse graph to manuscript seems feasible?
Line 64:		Line 64:
	* question: understanding different levels of value of having a knowledge graph for someone else who didn't create it		* question: understanding different levels of value of having a knowledge graph for someone else who didn't create it
	** --> Joel can add links to ongoing lit review on this question - not resolved yet		** --> Joel can add links to ongoing lit review on this question - not resolved yet
	* theme/idea: "compiling" from discourse to knowledge graphs ~~(entities as nodes, relations that are~~		* theme/idea: "compiling" from discourse to knowledge graphs
	** Aakanksha: often see ontologies in isolation		** Aakanksha: often see ontologies in isolation
	*** Aakanksha: can look into NLP around adding context to knowledge graphs		*** Aakanksha: can look into NLP around adding context to knowledge graphs
Line 70:		Line 70:
	** Grounding abstractions: https://www.susielu.com/data-viz/abstractions		** Grounding abstractions: https://www.susielu.com/data-viz/abstractions

			== Contextualizing knowledge graphs ==

			== Understanding knowledge graph transfer ==

			== Compiling graphs to manuscripts ==

	== Next Steps ==		== Next Steps ==

Joelchan at 17:16, 12 November 2022

2022-11-12T17:16:33Z

@@ Line 21: / Line 21: @@
 * [[Simularium]] - https://simularium.allencell.org/
 * [[Vivarium Collective]] - https://vivarium-collective.github.io/
 == Next Steps ==

Khinsen: Add myself

2022-11-12T08:47:41Z

Add myself

← Older revision		Revision as of 08:47, 12 November 2022
Line 5:		Line 5:
	\|Discord Channel Name=#computable-graphs		\|Discord Channel Name=#computable-graphs
	\|Discord Channel URL=https://discord.com/channels/1029514961782849607/1038983137222467604		\|Discord Channel URL=https://discord.com/channels/1029514961782849607/1038983137222467604
	\|Members=Akila Wijerathna-Yapa, Deniz Aydemir, Eran Egmon, Joel Chan, Matthew Akamatsu, Michael Gartner, ~~Aakanksha Naik~~		\|Members=Aakanksha Naik, Akila Wijerathna-Yapa, Deniz Aydemir, Eran Egmon, Joel Chan, Matthew Akamatsu, Michael Gartner, Konrad Hinsen
	}}		}}

	Facilitator/Point of Contact: [[Has Facilitator::Joel Chan]]		Facilitator/Point of Contact: [[Has Facilitator::Joel Chan]]