# Medical NLP and Utility API

This API primarily wraps others with the [Zensols Framework] to provide easy
way and reproducible method of utilization and experimentation with medical and
clinical natural language text.  It provides the following functionality:

* [UMLS Access via UTS]
* [Medical Concept and Entity Linking]
* [Using CUI as Word Embeddings](#using-cui-as-word-embeddings)
* [Entity Linking with cTAKES](#entity-linking-with-ctakes)

The rest of this document is structured as a cookbook style tutorial.  Each
sub-section describes the examples in the [examples] directory.

**Important**: many of the examples use [UMLS] UTS service, which requires a
key that is provided by NIH.  If you do not have a key, request one and add it
to the [UTS key file].


## Medical Concept and Entity Linking

Concept linking with [CUIs] is provided using the same interface as the
[Zensols NLP parsing API].  The resource library provided with this package
creates a `mednlp_doc_parser` as shown in the [entity-example].  First we start
with the configuration with file name `features.conf`, which starts with
telling the [CLI] to import the [Zensols NLP package] and this
(`zensols.mednlp`) package:
```ini
[import]
sections = list: imp_conf

[imp_conf]
type = importini
config_files = list:
    resource(zensols.nlp): resources/obj.conf,
    resource(zensols.nlp): resources/mapper.conf,
    resource(zensols.mednlp): resources/lang.conf
```

Next configure the parser with specific features, since otherwise, the parser
will retain all medical and non-medical features:
```ini
[mednlp_doc_parser]
token_feature_ids = set: norm, is_ent, cui, cui_, pref_name_, detected_name_, is_concept, ent_, ent
```

Finally, declare the application, which is needed by the [CLI] glue code to
invoke the class we will write afterward:
```ini
[app]
class_name = ${program:name}.Application
doc_parser = instance: mednlp_doc_parser
```

Next comes the application class:
```python
@dataclass
class Application(object):
    doc_parser: FeatureDocumentParser = field()

    def show(self, sent: str = None):
        if sent is None:
            sent = 'He was diagnosed with kidney failure in the United States.'
        doc: FeatureDocument = self.doc_parser(sent)
        print('first three tokens:')
        for tok in it.islice(doc.token_iter(), 3):
            print(tok.norm)
            tok.write_attributes(1, include_type=False)
```
This uses the document parser to create the feature document, which has both
the medical and linguistic features in tokens (provided by `token_iter()`) of the document.

Use the [CLI] API in the entry point to use the configuration and application
class:
```python
if (__name__ == '__main__'):
    CliHarness(
        app_config_resource='uts.conf',
        app_config_context=ProgramNameConfigurator(
            None, default='uts').create_section(),
        proto_args='',
    ).run()
```

Running the program produces one such token data:
```
...
diagnosed
    cui=11900
    cui_=C0011900
    detected_name_=diagnosed
    ent=13188083023294932426
    ent_=concept
    i=2
    i_sent=2
    idx=7
    is_concept=True
    is_ent=True
    norm=diagnosed
    pref_name_=Diagnosis
...
```
See the full [entity example] for the full example code, which will also output
both linguistic and medical features as a [Pandas] data frame.


## UMLS Access via UTS

NIH provides a very rough REST client using the `requests` library given as an
example.  This API takes that example, adds some "rigor" and structure in a
an easy to use class called `UTSClient`.  This is configured by first defining
paths for where fetched entities are cached:
```ini
[default]
# root directory given by the application, which is the parent directory
root_dir = ${appenv:root_dir}/..
# the directory to hold the cached UMLS data
cache_dir = ${root_dir}/cache
```

Next, import the this package's resource library (`zensols.mednlp`).  Note we
have to refer to sections that substitute the `default` section's data:
```ini
[import]
references = list: uts, default
sections = list: imp_uts_key, imp_conf

[imp_conf]
type = importini
config_file = resource(zensols.mednlp): resources/uts.conf

[imp_uts_key]
type = json
default_section = uts
config_file = ${default:root_dir}/uts-key.json
```
The `imp_uts_key` points to a file where you put add your UTS key, which is
given by NIH.

Now indicate where to cache the [UMLS] data and define our application we'll
write afterward:
```ini
# UTS (UMLS access)
[uts]
cache_file = ${default:cache_dir}/uts-request.dat
```

For brevity the [CLI] application code and configuration is omitted (see [UMLS
Access via UTS] for more detail).

To use the API to first search a term, then print entity information, we can
use the `search_term` method with `get_atoms`:
```python
@dataclass
class Application(object):
    ...
    def lookup(self, term: str = 'heart'):
        # terms are returned as a list of pages with dictionaries of data
        pages: List[Dict[str, str]] = self.uts_client.search_term(term)
        # get all term dictionaries from the first page
        terms: Dict[str, str] = pages[0]
        # get the concept unique identifier
        cui: str = terms['ui']

        # print atoms of this concept
        print('atoms:')
        pprint(self.uts_client.get_atoms(cui))
```
This yields the following output:
```
atoms:
{'ancestors': None,
 'classType': 'Atom',
 'code': 'https://uts-ws.nlm.nih.gov/rest/content/2020AA/source/MTH/NOCODE',
 'concept': 'https://uts-ws.nlm.nih.gov/rest/content/2020AA/CUI/C0018787',
 'contentViewMemberships': [{'memberUri': 'https://uts-ws.nlm.nih.gov/rest/content-views/2020AA/CUI/C1700357/member/A0066369',
                             'name': 'MetaMap NLP View',
                             'uri': 'https://uts-ws.nlm.nih.gov/rest/content-views/2020AA/CUI/C1700357'}],
 'name': 'Heart',
 'obsolete': 'false',
 'rootSource': 'MTH',
...
}
```

See the full [UTS example] for the full example code.


## Using CUI as Word Embeddings

[cui2vec] was trained and can be in the same way as [word2vec].  Such examples
is computing a similarity between [UMLS] [CUIs].  This API provides access to
the vectors directly along with all the functionality using [cui2vec] with the
[gensim] package.  This example computes the similarity between two medical
concepts.  For brevity the [CLI] application code and configuration is omitted
(see [UMLS Access via UTS] for more detail).

Let's jump right to how we import everything we need for the [cui2vec] example,
which the `uts` and `cui2vec` resource libraries:
```ini
[imp_conf]
type = importini
config_files = list:
    resource(zensols.mednlp): resources/uts.conf,
    resource(zensols.mednlp): resources/cui2vec.conf
```
The UTS configuration is given as in the [UMLS Access via UTS] section and the
parser is configured as in the [Medical Concept and Entity Linking] section.

With the high level classes given the configuration is class looks similar to
what we've seen before, this time we define a `similarity` method/[CLI] action:
```python
@dataclass
class Application(object):
    def similarity(self, term: str = 'heart disease', topn: int = 5):
```

Next, get the [gensim] `KeyedVectors` instance, which provides (among *many*
other useful methods) one to compute the similarity between two words, or in
our case, two medical [CUIs]:
```python
        embedding: Cui2VecEmbedModel = self.cui2vec_embedding
        kv: KeyedVectors = embedding.keyed_vectors
```

Next we use UTS to get the term we're searching on, use [gensim] to find
similarities, and output them:
```python
        res: List[Dict[str, str]] = self.uts_client.search_term(term)
        cui: str = res[0]['ui']
        sims_by_word: List[Tuple[str, float]] = kv.similar_by_word(cui, topn)
        for rel_cui, proba in sims_by_word:
            rel_atom: Dict[str, str] = self.uts_client.get_atoms(rel_cui)
            rel_name = rel_atom.get('name', 'Unknown')
            print(f'{rel_name} ({rel_cui}): {proba * 100:.2f}%')
```

The output contains the top (`topn`) 5 matches and their similarity to the
search term in the example `heart`:
```
Heart failure (C0018801): 72.03%
Atrial Premature Complexes (C0033036): 71.53%
Chronic myocardial ischemia (C0264694): 69.68%
Right bundle branch block (C0085615): 69.34%
First degree atrioventricular block (C0085614): 69.09%
```

See the full [cui2vec example] for the full example code.


## Entity Linking with cTAKES

This package provides an interface to [cTAKES], which primarily manages the
file system and invokes the Java program to produce results.  It then uses the
[ctakes-parser] to create a data frame of features and linked entities from
tokens of the source text.

The configuration is a bit more involved since you have to indicate where the
[cTAKES] program is installed, and provide your NIH key as detailed in the
[UMLS Access via UTS] section:
```ini
[import]
# refer to sections for which we need substitution in this file
references = list: default, ctakes, uts
sections = list: imp_env, imp_uts_key, imp_conf

# expose the user HOME environment variable
[imp_env]
type = environment
section_name = env
includes = set: HOME

# import the Zensols NLP UTS resource library
[imp_conf]
type = importini
config_files = list:
    resource(zensols.mednlp): resources/uts.conf,
    resource(zensols.mednlp): resources/ctakes.conf

# indicate where Apache cTAKES is installed
[ctakes]
home = ${env:home}/opt/app/ctakes-4.0.0.1
source_dir = ${default:cache_dir}/ctakes/source
```
For brevity the [CLI] application code and configuration is omitted, and other
configuration given in previous sections (see [UMLS Access via UTS] for more
detail). See the full [cTAKES example] for the full example code.

The pertinent snippet to get the [Pandas] data frame from the medical text is
very simple:
```python
@dataclass
class Application(object):
    def entities(self, sent: str = None, output: Path = None):
        if sent is None:
            sent = 'He was diagnosed with kidney failure in the United States.'
        self.ctakes_stash.set_documents([sent])
        df: pd.DataFrame = self.ctakes_stash['0']
        print(df)
        if output is not None:
            df.to_csv(output)
            print(f'wrote: {output}')
```
The `set_documents` expects a list of text, which is saved to disk.  When
[cTAKES] is run, the directory where this list of text is saved (one file per
element in the list).  The access to the [Stash] accesses the first document by
element ID.  **Note**: the element ID has to be a string to follow the [Stash]
API.


<!-- links -->
[UMLS Access via UTS]: #umls-access-via-uts
[Medical Concept and Entity Linking]: #medical-concept-and-entity-linking

[UMLS]: https://www.nlm.nih.gov/research/umls/
[CUIs]: https://www.nlm.nih.gov/research/umls/new_users/online_learning/Meta_005.html
[cui2vec]: https://arxiv.org/abs/1804.01486
[word2vec]: https://arxiv.org/abs/1301.3781

[Pandas]: https://pandas.pydata.org
[gensim]: https://radimrehurek.com/gensim/
[cTAKES]: https://ctakes.apache.org
[ctakes-parser]: https://pypi.org/project/ctakes-parser

[Zensols Framework]: https://arxiv.org/abs/2109.03383
[CLI]: https://plandes.github.io/util/doc/command-line.html
[Stash]: https://plandes.github.io/util/api/zensols.persist.html#zensols.persist.domain.Stash
[Zensols NLP package]: https://github.com/plandes/nlparse
[Zensols NLP parsing API]: https://plandes.github.io/nlparse/doc/feature-doc.html

[examples]: https://github.com/plandes/mednlp/tree/master/example
[entity example]: https://github.com/plandes/mednlp/tree/master/example/features
[cTAKES example]: https://github.com/plandes/mednlp/tree/master/example/ctakes
[cui2vec example]: https://github.com/plandes/mednlp/tree/master/example/cui2vec
[UTS example]: https://github.com/plandes/mednlp/tree/master/example/uts
[UTS key file]: https://github.com/plandes/mednlp/tree/master/example/uts-key.json