NLU release notes

 

NLU Version 4.0.0

OCR Visual Tables into Pandas DataFrames from PDF/DOC(X)/PPT files, 1000+ new state-of-the-art transformer models for Question Answering (QA) for over 30 languages, up to 700% speedup on GPU, 20 Biomedical models for over 8 languages, 50+ Terminology Code Mappers between RXNORM, NDC, UMLS,ICD10, ICDO, UMLS, SNOMED and MESH, Deidentification in Romanian, various Spark NLP helper methods and much more in 1 line of code with John Snow Labs NLU 4.0.0


NLU 4.0 for OCR Overview

On the OCR side, we now support extracting tables from PDF/DOC(X)/PPT files into structured pandas dataframe, making it easier than ever before to analyze bulks of files visually!

Checkout the OCR Tutorial for extracting Tables from Image/PDF/DOC(X) files Open In Colab to see this in action

These models grab all Table data from the files detected and return a list of Pandas DataFrames,
containing Pandas DataFrame for every table detected

NLU Spell Transformer Class
nlu.load(pdf2table) PdfToTextTable
nlu.load(ppt2table) PptToTextTable
nlu.load(doc2table) DocToTextTable

This is powerd by John Snow Labs Spark OCR Annotataors for PdfToTextTable, DocToTextTable, PptToTextTable


NLU 4.0 Core Overview

  • On the NLU core side we have over 1000+ new state-of-the-art models in over 30 languages for modern extractive transformer-based Question Answering problems powerd by the ALBERT/BERT/DistilBERT/DeBERTa/RoBERTa/Longformer Spark NLP Annotators trained on various SQUAD-like QA datasets for domains like Twitter, Tech, News, Biomedical COVID-19 and in various model subflavors like sci_bert, electra, mini_lm, covid_bert, bio_bert, indo_bert, muril, sapbert, bioformer, link_bert, mac_bert

  • Additionally up to 700% speedup transformer-based Word Embeddings on GPU and up to 97% speedup on CPU for tensorflow operations, support for Apple M1 chips, Pyspark 3.2 and 3.3 support. Ontop of this, we are now supporting Apple M1 based architectures and every Pyspark 3.X version, while deprecating support for Pyspark 2.X.

  • Finally, NLU-Core features various new helper methods for working with Spark NLP and embellishes now the entire universe of Annotators defined by Spark NLP and Spark NLP for healthcare.


NLU 4.0 for Healthcare Overview

  • On the healthcare side NLU features 20 Biomedical models for over 8 languages (English, French, Italian, Portuguese, Romanian, Catalan and Galician) detect entities like HUMAN and SPECIES based on LivingNER corpus

  • Romanian models for Deidentification and extracting Medical entities like MeasurementsFormSymptomRouteProcedureDisease_Syndrome_DisorderScoreDrug_IngredientPulseFrequencyDateBody_PartDrug_Brand_NameTimeDirectionDosageMedical_DeviceImaging_TechniqueTestImaging_FindingsImaging_TestTest_ResultWeightClinical_Dept and Units with SPELL and SPELL respectively

  • English NER models for parsing entities in Clinical Trial Abstracts like Age, AllocationRatio, Author, BioAndMedicalUnit, CTAnalysisApproach, CTDesign, Confidence, Country, DisorderOrSyndrome, DoseValue, Drug, DrugTime, Duration, Journal, NumberPatients, PMID, PValue, PercentagePatients, PublicationYear, TimePoint, Value using en.med_ner.clinical_trials_abstracts.pipe and also Pathogen NER models for PathogenMedicalConditionMedicine with en.med_ner.pathogen and GENE_PROTEIN with en.med_ner.biomedical_bc2gm.pipeline

  •  First Public Health Model for Emotional Stress classification It is a PHS-BERT-based model and trained with the Dreaddit dataset using en.classify.stress

  • 50 + new Entity Mappers for problems like :

    • Extract section headers in scientific articles and normalize them with en.map_entity.section_headers_normalized
    • Map medical abbreviates to their definitions with en.map_entity.abbreviation_to_definition
    • Map drugs to action and treatments with en.map_entity.drug_to_action_treatment
    • Map drug brand to their National Drug Code (NDC) with en.map_entity.drug_brand_to_ndc
    • Convert between terminologies using en.<START_TERMINOLOGY>_to_<TARGET_TERMINOLOGY>
      • This works for the terminologies rxnorm, ndc, umls, icd10cm, icdo, umls, snomed, mesh
        • snomed_to_icdo
        • snomed_to_icd10cm
        • rxnorm_to_umls
    • powerd by Spark NLP for Healthcares ChunkMapper Annotator

Extract Tables from PDF files as Pandas DataFrames

Sample PDF: Sample PDF

nlu.load('pdf2table').predict('/path/to/sample.pdf')  

Output of PDF Table OCR :

mpg cyl disp hp drat wt qsec vs am gear
21 6 160 110 3.9 2.62 16.46 0 1 4
21 6 160 110 3.9 2.875 17.02 0 1 4
22.8 4 108 93 3.85 2.32 18.61 1 1 4
21.4 6 258 110 3.08 3.215 19.44 1 0 3
18.7 8 360 175 3.15 3.44 17.02 0 0 3
13.3 8 350 245 3.73 3.84 15.41 0 0 3
19.2 8 400 175 3.08 3.845 17.05 0 0 3
27.3 4 79 66 4.08 1.935 18.9 1 1 4
26 4 120.3 91 4.43 2.14 16.7 0 1 5
30.4 4 95.1 113 3.77 1.513 16.9 1 1 5
15.8 8 351 264 4.22 3.17 14.5 0 1 5
19.7 6 145 175 3.62 2.77 15.5 0 1 5
15 8 301 335 3.54 3.57 14.6 0 1 5
21.4 4 121 109 4.11 2.78 18.6 1 1 4

Extract Tables from DOC/DOCX files as Pandas DataFrames

Sample DOCX: Sample DOCX

nlu.load('doc2table').predict('/path/to/sample.docx')  

Output of DOCX Table OCR :

Screen Reader Responses Share
JAWS 853 49%
NVDA 238 14%
Window-Eyes 214 12%
System Access 181 10%
VoiceOver 159 9%

Extract Tables from PPT files as Pandas DataFrame

Sample PPT with two tables: Sample PPT with two tables

nlu.load('ppt2table').predict('/path/to/sample.docx')  

Output of PPT Table OCR :

Sepal.Length Sepal.Width Petal.Length Petal.Width Species
5.1 3.5 1.4 0.2 setosa
4.9 3 1.4 0.2 setosa
4.7 3.2 1.3 0.2 setosa
4.6 3.1 1.5 0.2 setosa
5 3.6 1.4 0.2 setosa
5.4 3.9 1.7 0.4 setosa

and

Sepal.Length Sepal.Width Petal.Length Petal.Width Species
6.7 3.3 5.7 2.5 virginica
6.7 3 5.2 2.3 virginica
6.3 2.5 5 1.9 virginica
6.5 3 5.2 2 virginica
6.2 3.4 5.4 2.3 virginica
5.9 3 5.1 1.8 virginica

Span Classifiers for question answering

Albert, Bert, DeBerta, DistilBert, LongFormer, RoBerta, XlmRoBerta based Transformer Architectures are now avaiable for question answering with almost 1000 models avaiable for 35 unique languages powerd by their corrosponding Spark NLP XXXForQuestionAnswering Annotator Classes and in various tuning and dataset flavours.

<lang>.answer_question.<domain>.<datasets>.<annotator_class><tune info>.by_<username> If multiple datasets or tune parameters are defined , they are connected with a _ .

These substrings define up the <domain> part of the NLU reference

These substrings define up the <dataset> part of the NLU reference

These substrings define up the <dataset> part of the NLU reference

These substrings define the <annotator_class> substring, if it does not map to a sparknlp annotator

These substrings define the <tune_info> substring, if it does not map to a sparknlp annotator

  • Train tweaks : multilingual,mini_lm,xtremedistiled,distilled,xtreme,augmented,zero_shot
  • Size tweaks xl, xxl, large, base, medium, base, small, tiny, cased, uncased
  • Dimension tweaks : 1024d,768d,512d,256d,128d,64d,32d

QA DataFormat

You need to use one of the Data formats below to pass context and question correctly to the model.


# use ||| to seperate question||context
data = 'What is my name?|||My name is Clara and I live in Berkeley'

# pass a tuple (question,context)
data = ('What is my name?','My name is Clara and I live in Berkeley')

# use pandas Dataframe, one column = question, one column=context
data = pd.DataFrame({
					 'question': ['What is my name?'],
					 'context': ["My name is Clara and I live in Berkely"]
					 })


# Get your answers with any of above formats 
nlu.load("en.answer_question.squadv2.deberta").predict(data)

returns :

answer answer_confidence context question
Clara 0.994931 My name is Clara and I live in Berkely What is my name?

New NLU helper Methods

You can see all features showcased in the Open In Colab notebook or on the new docs page for Spark NLP utils

nlu.viz(pipe,data)

Visualize input data with an already configured Spark NLP pipeline,
for Algorithms of type (Ner,Assertion, Relation, Resolution, Dependency)
using Spark NLP Display
Automatically infers applicable viz type and output columns to use for visualization.
Example:

# works with Pipeline, LightPipeline, PipelineModel,PretrainedPipeline List[Annotator]
ade_pipeline = PretrainedPipeline('explain_clinical_doc_ade', 'en', 'clinical/models')

text = """I have an allergic reaction to vancomycin.
My skin has be itchy, sore throat/burning/itchy, and numbness in tongue and gums.
I would not recommend this drug to anyone, especially since I have never had such an adverse reaction to any other medication."""

nlu.viz(ade_pipeline, text)

returns:

If a pipeline has multiple models candidates that can be used for a viz,
the first Annotator that is vizzable will be used to create viz.
You can specify which type of viz to create with the viz_type parameter

Output columns to use for the viz are automatically deducted from the pipeline, by using the first annotator that provides the correct output type for a specific viz.
You can specify which columns to use for a viz by using the
corresponding ner_col, pos_col, dep_untyped_col, dep_typed_col, resolution_col, relation_col, assertion_col, parameters.

nlu.autocomplete_pipeline(pipe)

Auto-Complete a pipeline or single annotator into a runnable pipeline by harnessing NLU’s DAG Autocompletion algorithm and returns it as NLU pipeline. The standard Spark pipeline is avaiable on the .vanilla_transformer_pipe attribute of the returned nlu pipe

Every Annotator and Pipeline of Annotators defines a DAG of tasks, with various dependencies that must be satisfied in topoligical order. NLU enables the completion of an incomplete DAG by finding or creating a path between the very first input node which is almost always is DocumentAssembler/MultiDocumentAssembler and the very last node(s), which is given by the topoligical sorting the iterable annotators parameter. Paths are created by resolving input features of annotators to the corrrosponding providers with matching storage references.

Example:

# Lets autocomplete the pipeline for a RelationExtractionModel, which as many input columns and sub-dependencies.
from sparknlp_jsl.annotator import RelationExtractionModel
re_model = RelationExtractionModel().pretrained("re_ade_clinical", "en", 'clinical/models').setOutputCol('relation')

text = """I have an allergic reaction to vancomycin.
My skin has be itchy, sore throat/burning/itchy, and numbness in tongue and gums.
I would not recommend this drug to anyone, especially since I have never had such an adverse reaction to any other medication."""

nlu_pipe = nlu.autocomplete_pipeline(re_model)
nlu_pipe.predict(text)

returns :

relation relation_confidence relation_entity1 relation_entity2 relation_entity2_class
1 1 allergic reaction vancomycin Drug_Ingredient
1 1 skin itchy Symptom
1 0.99998 skin sore throat/burning/itchy Symptom
1 0.956225 skin numbness Symptom
1 0.999092 skin tongue External_body_part_or_region
0 0.942927 skin gums External_body_part_or_region
1 0.806327 itchy sore throat/burning/itchy Symptom
1 0.526163 itchy numbness Symptom
1 0.999947 itchy tongue External_body_part_or_region
0 0.994618 itchy gums External_body_part_or_region
0 0.994162 sore throat/burning/itchy numbness Symptom
1 0.989304 sore throat/burning/itchy tongue External_body_part_or_region
0 0.999969 sore throat/burning/itchy gums External_body_part_or_region
1 1 numbness tongue External_body_part_or_region
1 1 numbness gums External_body_part_or_region
1 1 tongue gums External_body_part_or_region

nlu.to_pretty_df(pipe,data)

Annotates a Pandas Dataframe/Pandas Series/Numpy Array/Spark DataFrame/Python List strings /Python String
with given Spark NLP pipeline, which is assumed to be complete and runnable and returns it in a pythonic pandas dataframe format.

Example:

# works with Pipeline, LightPipeline, PipelineModel,PretrainedPipeline List[Annotator]
ade_pipeline = PretrainedPipeline('explain_clinical_doc_ade', 'en', 'clinical/models')

text = """I have an allergic reaction to vancomycin.
My skin has be itchy, sore throat/burning/itchy, and numbness in tongue and gums.
I would not recommend this drug to anyone, especially since I have never had such an adverse reaction to any other medication."""

# output is same as nlu.autocomplete_pipeline(re_model).nlu_pipe.predict(text)
nlu.to_pretty_df(ade_pipeline,text)

returns :

assertion asserted_entitiy entitiy_class assertion_confidence
present allergic reaction ADE 0.998
present itchy ADE 0.8414
present sore throat/burning/itchy ADE 0.9019
present numbness in tongue and gums ADE 0.9991

Annotators are grouped internally by NLU into output levels token,sentence, document,chunk and relation Same level annotators output columns are zipped and exploded together to create the final output df. Additionally, most keys from the metadata dictionary in the result annotations will be collected and expanded into their own columns in the resulting Dataframe, with special handling for Annotators that encode multiple metadata fields inside of one, seperated by strings like ||| or :::. Some columns are omitted from metadata to reduce total amount of output columns, these can be re-enabled by setting metadata=True

For a given pipeline output level is automatically set to the last anntators output level by default. This can be changed by defining to_preddty_df(pipe,text,output_level='my_level' for levels token,sentence, document,chunk and relation .

nlu.to_nlu_pipe(pipe)

Convert a pipeline or list of annotators into a NLU pipeline making .predict() and .viz() avaiable for every Spark NLP pipeline. Assumes the pipeline is already runnable.

# works with Pipeline, LightPipeline, PipelineModel,PretrainedPipeline List[Annotator]
ade_pipeline = PretrainedPipeline('explain_clinical_doc_ade', 'en', 'clinical/models')

text = """I have an allergic reaction to vancomycin.
My skin has be itchy, sore throat/burning/itchy, and numbness in tongue and gums.
I would not recommend this drug to anyone, especially since I have never had such an adverse reaction to any other medication."""

nlu_pipe = nlu.to_nlu_pipe(ade_pipeline)

# Same output as nlu.to_pretty_df(pipe,text) 
nlu_pipe.predict(text)

# same output as nlu.viz(pipe,text)
nlu_pipe.viz(text)

# Acces auto-completed Spark NLP big data pipeline,
nlu_pipe.vanilla_transformer_pipe.transform(spark_df)

returns :

assertion asserted_entitiy entitiy_class assertion_confidence
present allergic reaction ADE 0.998
present itchy ADE 0.8414
present sore throat/burning/itchy ADE 0.9019
present numbness in tongue and gums ADE 0.9991

and


NLU captures every Annotator of Spark NLP and Spark NLP for healthcare

The entire universe of Annotators in Spark NLP and Spark-NLP for healthcare is now embellished by NLU Components by using generalizable annotation extractors methods and configs internally to support enable the new NLU util methods. The following annotator classes are newly captured:

  • AssertionFilterer
  • ChunkConverter
  • ChunkKeyPhraseExtraction
  • ChunkSentenceSplitter
  • ChunkFiltererApproach
  • ChunkFilterer
  • ChunkMapperApproach
  • ChunkMapperFilterer
  • DocumentLogRegClassifierApproach
  • DocumentLogRegClassifierModel
  • ContextualParserApproach
  • ReIdentification
  • NerDisambiguator
  • NerDisambiguatorModel
  • AverageEmbeddings
  • EntityChunkEmbeddings
  • ChunkMergeApproach
  • ChunkMergeApproach
  • IOBTagger
  • NerChunker
  • NerConverterInternalModel
  • DateNormalizer
  • PosologyREModel
  • RENerChunksFilter
  • ResolverMerger
  • AnnotationMerger
  • Router
  • Word2VecApproach
  • WordEmbeddings
  • EntityRulerApproach
  • EntityRulerModel
  • TextMatcherModel
  • BigTextMatcher
  • BigTextMatcherModel
  • DateMatcher
  • MultiDateMatcher
  • RegexMatcher
  • TextMatcher
  • NerApproach
  • NerCrfApproach
  • NerOverwriter
  • DependencyParserApproach
  • TypedDependencyParserApproach
  • SentenceDetectorDLApproach
  • SentimentDetector
  • ViveknSentimentApproach
  • ContextSpellCheckerApproach
  • NorvigSweetingApproach
  • SymmetricDeleteApproach
  • ChunkTokenizer
  • ChunkTokenizerModel
  • RecursiveTokenizer
  • RecursiveTokenizerModel
  • Token2Chunk
  • WordSegmenterApproach
  • GraphExtraction
  • Lemmatizer
  • Normalizer

All NLU 4.0 for Healthcare Models

Some examples:

en.rxnorm.umls.mapping

Code:

nlu.load('en.rxnorm.umls.mapping').predict('1161611 315677')

mapped_entity_umls_code_origin_entity mapped_entity_umls_code
1161611 C3215948
315677 C0984912
en.ner.clinical_trials_abstracts

Code:

nlu.load('en.ner.clinical_trials_abstracts').predict('A one-year, randomised, multicentre trial comparing insulin glargine with NPH insulin in combination with oral agents in patients with type 2 diabetes.')

Results:

  entities_clinical_trials_abstracts entities_clinical_trials_abstracts_class entities_clinical_trials_abstracts_confidence
0 randomised CTDesign 0.9996
0 multicentre CTDesign 0.9998
0 insulin glargine Drug 0.99135
0 NPH insulin Drug 0.96875
0 type 2 diabetes DisorderOrSyndrome 0.999933

Code:

nlu.load('en.ner.clinical_trials_abstracts').viz('A one-year, randomised, multicentre trial comparing insulin glargine with NPH insulin in combination with oral agents in patients with type 2 diabetes.')

Results:

en.med_ner.pathogen

Code:

nlu.load('en.med_ner.pathogen').predict('Racecadotril is an antisecretory medication and it has better tolerability than loperamide. Diarrhea is the condition of having loose, liquid or watery bowel movements each day. Signs of dehydration often begin with loss of the normal stretchiness of the skin. While it has been speculated that rabies virus, Lyssavirus and Ephemerovirus could be transmitted through aerosols, studies have concluded that this is only feasible in limited conditions.')

Results:

  entities_pathogen entities_pathogen_class entities_pathogen_confidence
0 Racecadotril Medicine 0.9468
0 loperamide Medicine 0.9987
0 Diarrhea MedicalCondition 0.9848
0 dehydration MedicalCondition 0.6307
0 rabies virus Pathogen 0.95685
0 Lyssavirus Pathogen 0.9694
0 Ephemerovirus Pathogen 0.6917

Code:

nlu.load('en.med_ner.pathogen').viz('Racecadotril is an antisecretory medication and it has better tolerability than loperamide. Diarrhea is the condition of having loose, liquid or watery bowel movements each day. Signs of dehydration often begin with loss of the normal stretchiness of the skin. While it has been speculated that rabies virus, Lyssavirus and Ephemerovirus could be transmitted through aerosols, studies have concluded that this is only feasible in limited conditions.')

Results:

es.med_ner.living_species.roberta

Code:

nlu.load('es.med_ner.living_species.roberta').predict('Lactante varón de dos años. Antecedentes familiares sin interés. Antecedentes personales: Embarazo, parto y periodo neonatal normal. En seguimiento por alergia a legumbres, diagnosticado con diez meses por reacción urticarial generalizada con lentejas y garbanzos, con dieta de exclusión a legumbres desde entonces. En ésta visita la madre describe episodios de eritema en zona maxilar derecha con afectación ocular ipsilateral que se resuelve en horas tras la administración de corticoides. Le ha ocurrido en 5-6 ocasiones, en relación con la ingesta de alimentos previamente tolerados. Exploración complementaria: Cacahuete, ac(ige)19.2 Ku.arb/l. Resultados: Ante la sospecha clínica de Síndrome de Frey, se tranquiliza a los padres, explicándoles la naturaleza del cuadro y se cita para revisión anual.')

Results:

  entities_living_species entities_living_species_class entities_living_species_confidence
0 Lactante varón HUMAN 0.93175
0 familiares HUMAN 1
0 personales HUMAN 1
0 neonatal HUMAN 0.9997
0 legumbres SPECIES 0.9962
0 lentejas SPECIES 0.9988
0 garbanzos SPECIES 0.9901
0 legumbres SPECIES 0.9976
0 madre HUMAN 1
0 Cacahuete SPECIES 0.998
0 padres HUMAN 1

Code:

nlu.load('es.med_ner.living_species.roberta').viz('Lactante varón de dos años. Antecedentes familiares sin interés. Antecedentes personales: Embarazo, parto y periodo neonatal normal. En seguimiento por alergia a legumbres, diagnosticado con diez meses por reacción urticarial generalizada con lentejas y garbanzos, con dieta de exclusión a legumbres desde entonces. En ésta visita la madre describe episodios de eritema en zona maxilar derecha con afectación ocular ipsilateral que se resuelve en horas tras la administración de corticoides. Le ha ocurrido en 5-6 ocasiones, en relación con la ingesta de alimentos previamente tolerados. Exploración complementaria: Cacahuete, ac(ige)19.2 Ku.arb/l. Resultados: Ante la sospecha clínica de Síndrome de Frey, se tranquiliza a los padres, explicándoles la naturaleza del cuadro y se cita para revisión anual.')

Results:

All healthcare models added in NLU 4.0 :

Language NLU Reference Spark NLP Reference Task Annotator Class model_id
en en.map_entity.abbreviation_to_definition abbreviation_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.abbreviation_to_definition
en en.map_entity.abbreviation_to_definition abbreviation_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.abbreviation_to_definition
en en.map_entity.drug_to_action_treatment drug_action_treatment_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.drug_to_action_treatment
en en.map_entity.drug_to_action_treatment drug_action_treatment_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.drug_to_action_treatment
en en.map_entity.drug_to_action_treatment drug_action_treatment_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.drug_to_action_treatment
en en.map_entity.drug_brand_to_ndc drug_brandname_ndc_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.drug_brand_to_ndc
en en.map_entity.drug_brand_to_ndc drug_brandname_ndc_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.drug_brand_to_ndc
en en.map_entity.icd10cm_to_snomed icd10cm_snomed_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.icd10cm_to_snomed
en en.map_entity.icd10cm_to_umls icd10cm_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.icd10cm_to_umls
en en.map_entity.icdo_to_snomed icdo_snomed_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.icdo_to_snomed
en en.map_entity.mesh_to_umls mesh_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.mesh_to_umls
en en.map_entity.rxnorm_to_action_treatment rxnorm_action_treatment_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.rxnorm_to_action_treatment
en en.map_entity.rxnorm_to_action_treatment rxnorm_action_treatment_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.rxnorm_to_action_treatment
en en.map_entity.rxnorm_resolver rxnorm_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.rxnorm_resolver
en en.map_entity.rxnorm_resolver rxnorm_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.rxnorm_resolver
en en.map_entity.rxnorm_to_ndc rxnorm_ndc_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.rxnorm_to_ndc
en en.map_entity.rxnorm_to_ndc rxnorm_ndc_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.rxnorm_to_ndc
en en.map_entity.rxnorm_to_ndc rxnorm_ndc_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.rxnorm_to_ndc
en en.map_entity.rxnorm_to_umls rxnorm_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.rxnorm_to_umls
en en.map_entity.rxnorm_to_umls rxnorm_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.rxnorm_to_umls
en en.map_entity.snomed_to_icd10cm snomed_icd10cm_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.snomed_to_icd10cm
en en.map_entity.snomed_to_icdo snomed_icdo_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.snomed_to_icdo
en en.map_entity.snomed_to_umls snomed_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.snomed_to_umls
en en.map_entity.snomed_to_icd10cm snomed_icd10cm_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.snomed_to_icd10cm
en en.map_entity.icd10cm_to_snomed icd10cm_snomed_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.icd10cm_to_snomed
en en.map_entity.snomed_to_icdo snomed_icdo_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.snomed_to_icdo
en en.map_entity.icdo_to_snomed icdo_snomed_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.icdo_to_snomed
en en.map_entity.rxnorm_to_umls rxnorm_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.rxnorm_to_umls
en en.map_entity.rxnorm_to_umls rxnorm_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.rxnorm_to_umls
en en.map_entity.icd10cm_to_umls icd10cm_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.icd10cm_to_umls
en en.map_entity.mesh_to_umls mesh_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.mesh_to_umls
en en.map_entity.snomed_to_umls snomed_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.map_entity.snomed_to_umls
en en.map_entity.section_headers_normalized normalized_section_header_mapper Chunk Mapping PretrainedPipeline Chunk Mappingen.map_entity.section_headers_normalized
en en.map_entity.section_headers_normalized normalized_section_header_mapper Chunk Mapping PretrainedPipeline Chunk Mappingen.map_entity.section_headers_normalized
en en.map_entity.section_headers_normalized normalized_section_header_mapper Chunk Mapping PretrainedPipeline Chunk Mappingen.map_entity.section_headers_normalized
en en.icd10cm_to_snomed icd10cm_snomed_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.icd10cm_to_snomed
en en.icd10cm_to_umls icd10cm_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.icd10cm_to_umls
en en.icdo_to_snomed icdo_snomed_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.icdo_to_snomed
en en.mesh_to_umls mesh_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.mesh_to_umls
en en.rxnorm_to_umls rxnorm_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.rxnorm_to_umls
en en.rxnorm_to_umls rxnorm_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.rxnorm_to_umls
en en.snomed_to_icd10cm snomed_icd10cm_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.snomed_to_icd10cm
en en.snomed_to_icdo snomed_icdo_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.snomed_to_icdo
en en.snomed_to_umls snomed_umls_mapper Chunk Mapping ChunkMapperModel Chunk Mappingen.snomed_to_umls
en en.map_entity.icd10cm_to_snomed.pipe icd10cm_snomed_mapping Pipeline Healthcare PretrainedPipeline Pipeline Healthcareen.map_entity.icd10cm_to_snomed.pipe
en en.map_entity.snomed_to_icd10cm.pipe snomed_icd10cm_mapping Pipeline Healthcare PretrainedPipeline Pipeline Healthcareen.map_entity.snomed_to_icd10cm.pipe
en en.map_entity.snomed_to_icd10cm.pipe snomed_icd10cm_mapping Pipeline Healthcare PretrainedPipeline Pipeline Healthcareen.map_entity.snomed_to_icd10cm.pipe
en en.map_entity.icdo_to_snomed.pipe icdo_snomed_mapping Pipeline Healthcare PretrainedPipeline Pipeline Healthcareen.map_entity.icdo_to_snomed.pipe
en en.map_entity.snomed_to_icdo.pipe snomed_icdo_mapping Pipeline Healthcare PretrainedPipeline Pipeline Healthcareen.map_entity.snomed_to_icdo.pipe
en en.map_entity.rxnorm_to_ndc.pipe rxnorm_ndc_mapping Pipeline Healthcare PretrainedPipeline Pipeline Healthcareen.map_entity.rxnorm_to_ndc.pipe
en en.med_ner.pathogen.pipeline ner_pathogen_pipeline Pipeline Healthcare PretrainedPipeline Pipeline Healthcareen.med_ner.pathogen.pipeline
en en.med_ner.biomedical_bc2gm.pipeline ner_biomedical_bc2gm_pipeline Pipeline Healthcare PretrainedPipeline Pipeline Healthcareen.med_ner.biomedical_bc2gm.pipeline
ro ro.deid.clinical clinical_deidentification Pipeline Healthcare MedicalNerModel Pipeline Healthcarero.deid.clinical
en en.med_ner.clinical_trials_abstracts.pipe ner_clinical_trials_abstracts_pipeline Pipeline Healthcare PretrainedPipeline Pipeline Healthcareen.med_ner.clinical_trials_abstracts.pipe
en en.ner.clinical_trials_abstracts ner_clinical_trials_abstracts Named Entity Recognition MedicalNerModel Named Entity Recognitionen.ner.clinical_trials_abstracts
en en.med_ner.clinical_trials_abstracts bert_token_classifier_ner_clinical_trials_abstracts Named Entity Recognition MedicalBertForTokenClassifier Named Entity Recognitionen.med_ner.clinical_trials_abstracts
en en.med_ner.pathogen ner_pathogen Named Entity Recognition MedicalNerModel Named Entity Recognitionen.med_ner.pathogen
en en.med_ner.living_species.token_bert bert_token_classifier_ner_living_species Named Entity Recognition MedicalBertForTokenClassifier Named Entity Recognitionen.med_ner.living_species.token_bert
en en.med_ner.living_species ner_living_species Named Entity Recognition MedicalNerModel Named Entity Recognitionen.med_ner.living_species
en en.med_ner.living_species.biobert ner_living_species_biobert Named Entity Recognition MedicalNerModel Named Entity Recognitionen.med_ner.living_species.biobert
en en.classify.stress bert_sequence_classifier_stress Text Classification MedicalBertForSequenceClassification Text Classificationen.classify.stress
es es.embed.scielo300d embeddings_scielo_300d Embeddings WordEmbeddingsModel Embeddingses.embed.scielo300d
es es.med_ner.living_species ner_living_species Named Entity Recognition MedicalNerModel Named Entity Recognitiones.med_ner.living_species
es es.med_ner.living_species.bert ner_living_species_bert Named Entity Recognition MedicalNerModel Named Entity Recognitiones.med_ner.living_species.bert
es es.med_ner.living_species.roberta ner_living_species_roberta Named Entity Recognition MedicalNerModel Named Entity Recognitiones.med_ner.living_species.roberta
es es.med_ner.living_species.300 ner_living_species_300 Named Entity Recognition MedicalNerModel Named Entity Recognitiones.med_ner.living_species.300
es es.med_ner.living_species ner_living_species Named Entity Recognition MedicalNerModel Named Entity Recognitiones.med_ner.living_species
fr fr.med_ner.living_species ner_living_species Named Entity Recognition MedicalNerModel Named Entity Recognitionfr.med_ner.living_species
fr fr.med_ner.living_species.bert ner_living_species_bert Named Entity Recognition MedicalNerModel Named Entity Recognitionfr.med_ner.living_species.bert
pt pt.med_ner.living_species.token_bert bert_token_classifier_ner_living_species Named Entity Recognition MedicalBertForTokenClassifier Named Entity Recognitionpt.med_ner.living_species.token_bert
pt pt.med_ner.living_species ner_living_species Named Entity Recognition MedicalNerModel Named Entity Recognitionpt.med_ner.living_species
pt pt.med_ner.living_species.roberta ner_living_species_roberta Named Entity Recognition MedicalNerModel Named Entity Recognitionpt.med_ner.living_species.roberta
pt pt.med_ner.living_species.bert ner_living_species_bert Named Entity Recognition MedicalNerModel Named Entity Recognitionpt.med_ner.living_species.bert
it it.med_ner.living_species ner_living_species Named Entity Recognition MedicalNerModel Named Entity Recognitionit.med_ner.living_species
it it.med_ner.living_species.bert ner_living_species_bert Named Entity Recognition MedicalNerModel Named Entity Recognitionit.med_ner.living_species.bert
it it.med_ner.living_species ner_living_species Named Entity Recognition MedicalNerModel Named Entity Recognitionit.med_ner.living_species
ca ca.med_ner.living_species ner_living_species Named Entity Recognition MedicalNerModel Named Entity Recognitionca.med_ner.living_species
gl gl.med_ner.living_species ner_living_species Named Entity Recognition MedicalNerModel Named Entity Recognitiongl.med_ner.living_species
ro ro.med_ner.living_species.bert ner_living_species_bert Named Entity Recognition MedicalNerModel Named Entity Recognitionro.med_ner.living_species.bert
ro ro.med_ner.clinical ner_clinical Named Entity Recognition MedicalNerModel Named Entity Recognitionro.med_ner.clinical
ro ro.embed.clinical.bert.base_cased ner_clinical_bert Named Entity Recognition MedicalNerModel Named Entity Recognitionro.embed.clinical.bert.base_cased
ro ro.med_ner.deid.subentity ner_deid_subentity Named Entity Recognition MedicalNerModel Named Entity Recognitionro.med_ner.deid.subentity
ro ro.med_ner.deid.subentity.bert ner_deid_subentity_bert Named Entity Recognition MedicalNerModel Named Entity Recognitionro.med_ner.deid.subentity.bert

All NLU 4.0 Core Models

All core models added in NLU 4.0 : Can be found on the NLU website because of Github Limitations

NLU Reference Spark NLP Reference Task Language Name(s) Annotator Class
bn.answer_question.tydiqa.multi_lingual_bert bert_qa_mbert_bengali_tydiqa_qa Question Answering Bengali BertForQuestionAnswering
es.answer_question.squadv2.electra.small electra_qa_biomedtra_small_es_squad2 Question Answering Castilian, Spanish BertForQuestionAnswering
es.answer_question.squad_sqac.bert.base_cased bert_qa_bert_base_spanish_wwm_cased_finetuned_sqac_finetuned_squad Question Answering Castilian, Spanish BertForQuestionAnswering
es.answer_question.squadv2.bert.base_cased.by_MMG bert_qa_bert_base_spanish_wwm_cased_finetuned_squad2_es_MMG Question Answering Castilian, Spanish BertForQuestionAnswering
es.answer_question.squadv2.bert.base_cased.by_mrm8488 bert_qa_bert_base_spanish_wwm_cased_finetuned_spa_squad2_es_mrm8488 Question Answering Castilian, Spanish BertForQuestionAnswering
es.answer_question.squadv2.bert.distilled_base_cased bert_qa_distill_bert_base_spanish_wwm_cased_finetuned_spa_squad2_es_mrm8488 Question Answering Castilian, Spanish BertForQuestionAnswering
es.answer_question.squad.ruperta.base.by_mrm8488 roberta_qa_RuPERTa_base_finetuned_squadv1 Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.squadv2.roberta.base roberta_qa_roberta_base_bne_squad2_hackathon_pln Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.squadv2_sqac.bert.base_cased_spa.by_MMG bert_qa_bert_base_spanish_wwm_cased_finetuned_spa_squad2_es_finetuned_sqac Question Answering Castilian, Spanish BertForQuestionAnswering
es.answer_question.squadv2_bio_medical.roberta.base roberta_qa_roberta_base_biomedical_es_squad2_hackathon_pln Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.squadv2_clinical_bio_medical.roberta.base roberta_qa_roberta_base_biomedical_clinical_es_squad2_hackathon_pln Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.squadv2_sqac.bert.base_cased.by_MMG bert_qa_bert_base_spanish_wwm_cased_finetuned_sqac_finetuned_squad2_es_MMG Question Answering Castilian, Spanish BertForQuestionAnswering
es.answer_question.squadv2_sqac.bert.base_cased_v2.by_MMG bert_qa_bert_base_spanish_wwm_cased_finetuned_squad2_es_finetuned_sqac Question Answering Castilian, Spanish BertForQuestionAnswering
es.answer_question.xlm_roberta.base xlm_roberta_qa_xlm_roberta_base_spanish Question Answering Castilian, Spanish XlmRoBertaForQuestionAnswering
es.answer_question.xlm_roberta.multilingual_large xlm_roberta_qa_xlm_roberta_large_qa_multilingual_finedtuned_ru_ru_AlexKay Question Answering Castilian, Spanish XlmRoBertaForQuestionAnswering
es.answer_question.squad.roberta.large.by_stevemobs roberta_qa_roberta_large_fine_tuned_squad_es_stevemobs Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.squadv2.roberta.base_v2 roberta_qa_RuPERTa_base_finetuned_squadv2 Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.squad.roberta.large.by_jamarju roberta_qa_roberta_large_bne_squad_2.0_es_jamarju Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.sqac.roberta.large.by_BSC-TeMU roberta_qa_BSC_TeMU_roberta_large_bne_sqac Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.squad.roberta.base.by_jamarju roberta_qa_roberta_base_bne_squad_2.0_es_jamarju Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.squad.roberta.base_4096.by_mrm8488 roberta_qa_longformer_base_4096_spanish_finetuned_squad Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.distil_bert.base_uncased distilbert_qa_distillbert_base_spanish_uncased_finetuned_qa_tar Question Answering Castilian, Spanish DistilBertForQuestionAnswering
es.answer_question.mlqa.distil_bert.base_uncased distilbert_qa_distillbert_base_spanish_uncased_finetuned_qa_mlqa Question Answering Castilian, Spanish DistilBertForQuestionAnswering
es.answer_question.sqac.bert.base bert_qa_beto_base_spanish_sqac Question Answering Castilian, Spanish BertForQuestionAnswering
es.answer_question.sqac.distil_bert.base_uncased distilbert_qa_distillbert_base_spanish_uncased_finetuned_qa_sqac Question Answering Castilian, Spanish DistilBertForQuestionAnswering
es.answer_question.sqac.roberta.base.by_BSC-TeMU roberta_qa_BSC_TeMU_roberta_base_bne_sqac Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.sqac.roberta.base.by_IIC roberta_qa_roberta_base_spanish_sqac Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.sqac.bert.base_cased bert_qa_bert_base_spanish_wwm_cased_finetuned_sqac Question Answering Castilian, Spanish BertForQuestionAnswering
es.answer_question.sqac.roberta.base.by_mrm8488 roberta_qa_mrm8488_roberta_base_bne_finetuned_sqac Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.sqac.roberta.base.by_nlp-en-es roberta_qa_nlp_en_es_roberta_base_bne_finetuned_sqac Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.sqac.roberta.large.by_PlanTL-GOB-ES roberta_qa_PlanTL_GOB_ES_roberta_large_bne_sqac Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.sqac.roberta.large.by_nlp-en-es roberta_qa_bertin_large_finetuned_sqac Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.squad.electra.small electra_qa_electricidad_small_finetuned_squadv1 Question Answering Castilian, Spanish BertForQuestionAnswering
es.answer_question.squad.roberta.base.by_IIC roberta_qa_roberta_base_spanish_squades Question Answering Castilian, Spanish RoBertaForQuestionAnswering
es.answer_question.sqac.roberta.base.by_PlanTL-GOB-ES roberta_qa_PlanTL_GOB_ES_roberta_base_bne_sqac Question Answering Castilian, Spanish RoBertaForQuestionAnswering
ch.answer_question.xlm_roberta xlm_roberta_qa_ADDI_CH_XLM_R Question Answering Chamorro XlmRoBertaForQuestionAnswering
da.answer_question.squad.bert bert_qa_danish_bert_botxo_qa_squad Question Answering Danish BertForQuestionAnswering
da.answer_question.squad.xlmr_roberta.base xlm_roberta_qa_xlmr_base_texas_squad_da_da_saattrupdan Question Answering Danish XlmRoBertaForQuestionAnswering
nl.answer_question.squadv2.bert.multilingual_base_cased bert_qa_bert_base_multilingual_cased_finetuned_dutch_squad2 Question Answering Dutch, Flemish BertForQuestionAnswering
en.answer_question.squad.roberta.large.by_csarron roberta_qa_roberta_large_squad_v1 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.large.by_rahulchakwate roberta_qa_roberta_large_finetuned_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.scibert.by_amoux bert_qa_scibert_nli_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.scibert.by_ixa-ehu bert_qa_SciBERT_SQuAD_QuAC Question Answering English BertForQuestionAnswering
en.answer_question.squad.scibert.uncased bert_qa_scibert_scivocab_uncased_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert bert_qa_spanbert_finetuned_squadv1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_1024d_seed_0 bert_qa_spanbert_base_cased_few_shot_k_1024_finetuned_squad_seed_0 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_1024d_seed_10 bert_qa_spanbert_base_cased_few_shot_k_1024_finetuned_squad_seed_10 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_1024d_seed_2 bert_qa_spanbert_base_cased_few_shot_k_1024_finetuned_squad_seed_2 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_1024d_seed_4 bert_qa_spanbert_base_cased_few_shot_k_1024_finetuned_squad_seed_4 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_1024d_seed_8 bert_qa_spanbert_base_cased_few_shot_k_1024_finetuned_squad_seed_8 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_1024d_seed_6 bert_qa_spanbert_base_cased_few_shot_k_1024_finetuned_squad_seed_6 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_128d_seed_10 bert_qa_spanbert_base_cased_few_shot_k_128_finetuned_squad_seed_10 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_128d_seed_4 bert_qa_spanbert_base_cased_few_shot_k_128_finetuned_squad_seed_4 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_128d_seed_6 bert_qa_spanbert_base_cased_few_shot_k_128_finetuned_squad_seed_6 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_128d_seed_8 bert_qa_spanbert_base_cased_few_shot_k_128_finetuned_squad_seed_8 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_256d_seed_10 bert_qa_spanbert_base_cased_few_shot_k_256_finetuned_squad_seed_10 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_32d_seed_0 bert_qa_spanbert_base_cased_few_shot_k_32_finetuned_squad_seed_0 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_32d_seed_10 bert_qa_spanbert_base_cased_few_shot_k_32_finetuned_squad_seed_10 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_32d_seed_2 bert_qa_spanbert_base_cased_few_shot_k_32_finetuned_squad_seed_2 Question Answering English BertForQuestionAnswering
en.answer_question.squad.roberta.distilled_base roberta_qa_distilroberta_base_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_1024d_seed_42 bert_qa_spanbert_base_cased_few_shot_k_1024_finetuned_squad_seed_42 Question Answering English BertForQuestionAnswering
en.answer_question.squad.roberta.distilled roberta_qa_distilroberta_finetuned_squadv1 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_scrambled_sq.by_huxxx657 roberta_qa_roberta_base_finetuned_scrambled_squad_5_new Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.by_sunitha roberta_qa_Roberta_Custom_Squad_DS Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_64d_seed_6 roberta_qa_roberta_base_few_shot_k_64_finetuned_squad_seed_6 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_64d_seed_8 roberta_qa_roberta_base_few_shot_k_64_finetuned_squad_seed_8 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_deletion_10.by_huxxx657 roberta_qa_roberta_base_finetuned_deletion_squad_10 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_deletion_15.by_huxxx657 roberta_qa_roberta_base_finetuned_deletion_squad_15 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_sae.by_jgammack roberta_qa_SAE_roberta_base_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_scrambled_10.by_huxxx657 roberta_qa_roberta_base_finetuned_scrambled_squad_10 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_scrambled_10_new.by_huxxx657 roberta_qa_roberta_base_finetuned_scrambled_squad_10_new Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_scrambled_15.by_huxxx657 roberta_qa_roberta_base_finetuned_scrambled_squad_15 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_scrambled_15_v2.by_huxxx657 roberta_qa_roberta_base_finetuned_scrambled_squad_15_new Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_scrambled_5.by_huxxx657 roberta_qa_roberta_base_finetuned_scrambled_squad_5 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.by_vuiseng9 roberta_qa_roberta_l_squadv1.1 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_32d_seed_6 bert_qa_spanbert_base_cased_few_shot_k_32_finetuned_squad_seed_6 Question Answering English BertForQuestionAnswering
en.answer_question.squad.roberta.base_seed_10 roberta_qa_roberta_base_few_shot_k_16_finetuned_squad_seed_10 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_seed_2 roberta_qa_roberta_base_few_shot_k_16_finetuned_squad_seed_2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_seed_4 roberta_qa_roberta_base_few_shot_k_16_finetuned_squad_seed_4 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_seed_42 roberta_qa_roberta_base_few_shot_k_16_finetuned_squad_seed_42 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_seed_6 roberta_qa_roberta_base_few_shot_k_16_finetuned_squad_seed_6 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_seed_8 roberta_qa_roberta_base_few_shot_k_16_finetuned_squad_seed_8 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_v1.by_huxxx657 roberta_qa_roberta_base_finetuned_squad_1 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_v2.by_huxxx657 roberta_qa_roberta_base_finetuned_squad_2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_v3.by_huxxx657 roberta_qa_roberta_base_finetuned_squad_3 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.by_cgou roberta_qa_fin_RoBERTa_v1_finetuned_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_seed_0 roberta_qa_roberta_base_few_shot_k_16_finetuned_squad_seed_0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_512d_seed_0 bert_qa_spanbert_base_cased_few_shot_k_512_finetuned_squad_seed_0 Question Answering English BertForQuestionAnswering
en.answer_question.squad_battery.bert.cased.by_batterydata bert_qa_batterybert_cased_squad_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_512d_seed_6 bert_qa_spanbert_base_cased_few_shot_k_512_finetuned_squad_seed_6 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.albert.xxl.by_replydotai albert_qa_xxlarge_v1_finetuned_squad2 Question Answering English AlbertForQuestionAnswering
en.answer_question.squadv2.albert.xxl.by_sultan albert_qa_BioM_xxlarge_SQuAD2 Question Answering English AlbertForQuestionAnswering
en.answer_question.squadv2.albert.xxl_512d albert_qa_xxlargev1_squad2_512 Question Answering English AlbertForQuestionAnswering
en.answer_question.squadv2.albert.xxl_v2 albert_qa_xxlarge_v2_squad2 Question Answering English AlbertForQuestionAnswering
en.answer_question.squadv2.bert.base bert_qa_bert_base_finetuned_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.base_cased.by_deepset bert_base_cased_qa_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.base_cased.by_vumichien bert_qa_tf_bert_base_cased_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.base_cased.by_ydshieh bert_qa_ydshieh_bert_base_cased_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.base_uncased.by_Vasanth bert_qa_bert_base_uncased_qa_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.base_uncased.by_deepset bert_qa_deepset_bert_base_uncased_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.base_uncased.by_twmkn9 bert_qa_twmkn9_bert_base_uncased_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.base_uncased_v2 bert_qa_bert_base_uncased_finetuned_squad_v2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.base_v2.by_mrm8488 bert_qa_bert_mini_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.base_v2_5.by_mrm8488 bert_qa_bert_mini_5_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.by_augustoortiz bert_qa_bert_finetuned_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.by_maroo93 bert_qa_squad2.0 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.by_pinecone bert_qa_bert_reader_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.distilled bert_qa_xdistil_l12_h384_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.distilled_medium bert_qa_bert_medium_squad2_distilled Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.large.by_Sindhu bert_qa_muril_large_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.large.by_phiyodr bert_qa_bert_large_finetuned_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.albert.xxl.by_elgeish albert_qa_cs224n_squad2.0_xxlarge_v1 Question Answering English AlbertForQuestionAnswering
en.answer_question.squadv2.albert.xl_v2 albert_qa_xlarge_v2_squad_v2 Question Answering English AlbertForQuestionAnswering
en.answer_question.squadv2.albert.large_v2 albert_qa_cs224n_squad2.0_large_v2 Question Answering English AlbertForQuestionAnswering
en.answer_question.squadv2.albert.base_v2.by_vumichien albert_qa_vumichien_base_v2_squad2 Question Answering English AlbertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_512d_seed_8 bert_qa_spanbert_base_cased_few_shot_k_512_finetuned_squad_seed_8 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_64d_seed_0 bert_qa_spanbert_base_cased_few_shot_k_64_finetuned_squad_seed_0 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_64d_seed_10 bert_qa_spanbert_base_cased_few_shot_k_64_finetuned_squad_seed_10 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_64d_seed_2 bert_qa_spanbert_base_cased_few_shot_k_64_finetuned_squad_seed_2 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_64d_seed_4 bert_qa_spanbert_base_cased_few_shot_k_64_finetuned_squad_seed_4 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_64d_seed_6 bert_qa_spanbert_base_cased_few_shot_k_64_finetuned_squad_seed_6 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_seed_42 bert_qa_spanbert_base_cased_few_shot_k_16_finetuned_squad_seed_42 Question Answering English BertForQuestionAnswering
en.answer_question.squad.xlm_roberta.by_jakobwes xlm_roberta_qa_xlm_roberta_squad_v1.1 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squad.xlm_roberta.by_meghana xlm_roberta_qa_hitalmqa_finetuned_squad Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squad_battery.bert.base_uncased bert_qa_batterydata_bert_base_uncased_squad_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.span_bert.base_cased_512d_seed_10 bert_qa_spanbert_base_cased_few_shot_k_512_finetuned_squad_seed_10 Question Answering English BertForQuestionAnswering
en.answer_question.squad.roberta.base_64d_seed_4 roberta_qa_roberta_base_few_shot_k_64_finetuned_squad_seed_4 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad_battery.bert.uncased.by_batterydata bert_qa_batterybert_uncased_squad_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad_battery.bert.uncased_only_bert.by_batterydata bert_qa_batteryonlybert_uncased_squad_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad_battery.scibert.cased bert_qa_batteryscibert_cased_squad_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad_battery.scibert.uncased bert_qa_batteryscibert_uncased_squad_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad_ben_tel.bert.by_krinal214 bert_qa_bert_all_squad_ben_tel_context Question Answering English BertForQuestionAnswering
en.answer_question.squad_covid.bert bert_qa_covid_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad_pubmed.biobert bert_qa_biobert_v1.1_pubmed_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad_translated.bert.by_krinal214 bert_qa_bert_all_squad_all_translated Question Answering English BertForQuestionAnswering
en.answer_question.squad_translated.bert.que.by_krinal214 bert_qa_bert_all_squad_que_translated Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.albert.base_v2.by_elgeish albert_qa_cs224n_squad2.0_base_v2 Question Answering English AlbertForQuestionAnswering
en.answer_question.squad_battery.bert.cased_only_bert.by_batterydata bert_qa_batteryonlybert_cased_squad_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.roberta.base_64d_seed_2 roberta_qa_roberta_base_few_shot_k_64_finetuned_squad_seed_2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_32d_seed_10 roberta_qa_roberta_base_few_shot_k_32_finetuned_squad_seed_10 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_64d_seed_0 roberta_qa_roberta_base_few_shot_k_64_finetuned_squad_seed_0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased_mtl.by_jgammack distilbert_qa_MTL_base_uncased_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased_sae.by_jgammack distilbert_qa_SAE_base_uncased_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased_v2.by_arvalinno distilbert_qa_base_uncased_finetuned_indosquad_v2 Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased_v2.by_ericRosello distilbert_qa_base_uncased_finetuned_squad_frozen_v2 Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased_v2.by_holtin distilbert_qa_holtin_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased_v2.by_huxxx657 distilbert_qa_base_uncased_finetuned_jumbling_squad_15 Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased_v3.by_anurag0077 distilbert_qa_anurag0077_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.by_AyushPJ distilbert_qa_test_squad_trained_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.by_ZYW distilbert_qa_test_squad_trained Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.by_abhilash1910 distilbert_qa_squadv1 Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.by_rowan1224 distilbert_qa_squad_slp Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.by_sunitha distilbert_qa_AQG_CV_Squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.by_tabo distilbert_qa_checkpoint_500_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.electra electra_qa_squad_slp Question Answering English BertForQuestionAnswering
en.answer_question.squad.electra.base.by_Palak electra_qa_google_base_discriminator_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.electra.base.by_mrm8488 electra_qa_base_finetuned_squadv1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.electra.base.by_usami electra_qa_base_discriminator_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.electra.base.by_valhalla electra_qa_base_discriminator_finetuned_squadv1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.electra.large.by_howey electra_qa_large_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.electra.small.by_Palak electra_qa_google_small_discriminator_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.electra.small.by_bdickson electra_qa_small_discriminator_finetuned_squad_1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased_full.by_holtin distilbert_qa_base_uncased_holtin_finetuned_full_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased_colab.by_Adrian distilbert_qa_base_uncased_finetuned_squad_colab Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_vkrishnamoorthy distilbert_qa_vkrishnamoorthy_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_vkmr distilbert_qa_vkmr_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_gokulkarthik distilbert_qa_gokulkarthik_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_graviraja distilbert_qa_graviraja_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_guhuawuli distilbert_qa_guhuawuli_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_hark99 distilbert_qa_hark99_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_hcy11 distilbert_qa_hcy11_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_hiiii23 distilbert_qa_hiiii23_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_holtin distilbert_qa_base_uncased_holtin_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_huggingfaceepita distilbert_qa_huggingfaceepita_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_huxxx657 distilbert_qa_huxxx657_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_jgammack distilbert_qa_jgammack_base_uncased_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.electra.small.by_hankzhong electra_qa_hankzhong_small_discriminator_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_jhoonk distilbert_qa_jhoonk_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_kaggleodin distilbert_qa_kaggleodin_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_lewtun distilbert_qa_base_uncased_finetuned_squad_v1 Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_machine2049 distilbert_qa_base_uncased_finetuned_squad_ Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_manudotc distilbert_qa_transformers_base_uncased_finetuneQA_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_sunitha distilbert_qa_base_uncased_3feb_2022_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_tli8hf distilbert_qa_unqover_base_uncased_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_tucan9389 distilbert_qa_tucan9389_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_uploaded by huggingface distilbert_qa_base_uncased_distilled_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_usami distilbert_qa_usami_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_vitusya distilbert_qa_vitusya_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_jsunster distilbert_qa_jsunster_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.roberta.base_64d_seed_10 roberta_qa_roberta_base_few_shot_k_64_finetuned_squad_seed_10 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.electra.small.by_mrm8488 electra_qa_small_finetuned_squadv1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.ixam_bert.by_MarcBrun bert_qa_ixambert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.roberta.base_128d_seed_42 roberta_qa_roberta_base_few_shot_k_128_finetuned_squad_seed_42 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_128d_seed_6 roberta_qa_roberta_base_few_shot_k_128_finetuned_squad_seed_6 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_128d_seed_8 roberta_qa_roberta_base_few_shot_k_128_finetuned_squad_seed_8 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_256d_seed_0 roberta_qa_roberta_base_few_shot_k_256_finetuned_squad_seed_0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_256d_seed_10 roberta_qa_roberta_base_few_shot_k_256_finetuned_squad_seed_10 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_256d_seed_2 roberta_qa_roberta_base_few_shot_k_256_finetuned_squad_seed_2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_256d_seed_4 roberta_qa_roberta_base_few_shot_k_256_finetuned_squad_seed_4 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_256d_seed_6 roberta_qa_roberta_base_few_shot_k_256_finetuned_squad_seed_6 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_256d_seed_8 roberta_qa_roberta_base_few_shot_k_256_finetuned_squad_seed_8 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_32d_seed_0 roberta_qa_roberta_base_few_shot_k_32_finetuned_squad_seed_0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.bert.large_tiny_768d.by_MichelBartels bert_qa_tinybert_6l_768d_squad2_large_teacher_finetuned Question Answering English BertForQuestionAnswering
en.answer_question.squad.roberta.base_32d_seed_2 roberta_qa_roberta_base_few_shot_k_32_finetuned_squad_seed_2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_32d_seed_4 roberta_qa_roberta_base_few_shot_k_32_finetuned_squad_seed_4 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_32d_seed_6 roberta_qa_roberta_base_few_shot_k_32_finetuned_squad_seed_6 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_32d_seed_8 roberta_qa_roberta_base_few_shot_k_32_finetuned_squad_seed_8 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_512d_seed_0 roberta_qa_roberta_base_few_shot_k_512_finetuned_squad_seed_0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_512d_seed_10 roberta_qa_roberta_base_few_shot_k_512_finetuned_squad_seed_10 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_512d_seed_2 roberta_qa_roberta_base_few_shot_k_512_finetuned_squad_seed_2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_512d_seed_4 roberta_qa_roberta_base_few_shot_k_512_finetuned_squad_seed_4 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_512d_seed_6 roberta_qa_roberta_base_few_shot_k_512_finetuned_squad_seed_6 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_512d_seed_8 roberta_qa_roberta_base_few_shot_k_512_finetuned_squad_seed_8 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_128d_seed_4 roberta_qa_roberta_base_few_shot_k_128_finetuned_squad_seed_4 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_128d_seed_2 roberta_qa_roberta_base_few_shot_k_128_finetuned_squad_seed_2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_128d_seed_10 roberta_qa_roberta_base_few_shot_k_128_finetuned_squad_seed_10 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_128d_seed_0 roberta_qa_roberta_base_few_shot_k_128_finetuned_squad_seed_0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.ixam_bert.eu_en_tunedby_MarcBrun bert_qa_ixambert_finetuned_squad_eu_en_MarcBrun Question Answering English BertForQuestionAnswering
en.answer_question.squad.ixam_bert.eu_tuned.by_MarcBrun bert_qa_ixambert_finetuned_squad_eu_MarcBrun Question Answering English BertForQuestionAnswering
en.answer_question.squad.link_bert.large bert_qa_linkbert_large_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.multi_lingual_bert.by_ZYW bert_qa_squad_mbert_model Question Answering English BertForQuestionAnswering
en.answer_question.squad.multi_lingual_bert.en_de_es.by_ZYW bert_qa_squad_mbert_en_de_es_model Question Answering English BertForQuestionAnswering
en.answer_question.squad.multi_lingual_bert.en_de_es_vi_zh.by_ZYW bert_qa_squad_mbert_en_de_es_vi_zh_model Question Answering English BertForQuestionAnswering
en.answer_question.squad.multi_lingual_bert.v2.by_ZYW bert_qa_squad_mbert_model_2 Question Answering English BertForQuestionAnswering
en.answer_question.squad.roberta.base.by_Firat roberta_qa_Firat_roberta_base_finetuned_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base.by_ahmedattia143 roberta_qa_roberta_squadv1_base Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base.by_csarron roberta_qa_roberta_base_squad_v1 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.electra.small_v2.by_bdickson electra_qa_small_discriminator_finetuned_squad_2 Question Answering English BertForQuestionAnswering
en.answer_question.squad.roberta.base.by_huxxx657 roberta_qa_huxxx657_roberta_base_finetuned_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base.by_mrm8488 roberta_qa_roberta_base_1B_1_finetuned_squadv1 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base.by_rahulchakwate roberta_qa_rahulchakwate_roberta_base_finetuned_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base.by_tli8hf roberta_qa_unqover_roberta_base_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_1024d_seed_0 roberta_qa_roberta_base_few_shot_k_1024_finetuned_squad_seed_0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_1024d_seed_10 roberta_qa_roberta_base_few_shot_k_1024_finetuned_squad_seed_10 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_1024d_seed_2 roberta_qa_roberta_base_few_shot_k_1024_finetuned_squad_seed_2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_1024d_seed_4 roberta_qa_roberta_base_few_shot_k_1024_finetuned_squad_seed_4 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_1024d_seed_42 roberta_qa_roberta_base_few_shot_k_1024_finetuned_squad_seed_42 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_1024d_seed_6 roberta_qa_roberta_base_few_shot_k_1024_finetuned_squad_seed_6 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base_1024d_seed_8 roberta_qa_roberta_base_few_shot_k_1024_finetuned_squad_seed_8 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.roberta.base.by_jgammack roberta_qa_roberta_base_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.bert.large_tiny_768d_v2.by_MichelBartels bert_qa_tinybert_6l_768d_squad2_large_teacher_finetuned_step1 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.tiny_768d bert_qa_tinybert_6l_768d_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.large_uncased.by_andi611 bert_qa_bert_large_uncased_whole_word_masking_squad2_with_ner_mit_restaurant_with_neg_with_repeat Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid.bert.uncased_4l_256d_a4a_256d bert_qa_bert_uncased_L_4_H_256_A_4_squad2_covid_qna Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid.bert.uncased_4l_512d_a8a_512d bert_qa_bert_uncased_L_4_H_512_A_8_squad2_covid_qna Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid.bert.uncased_4l_768d_a12a_768d bert_qa_bert_uncased_L_4_H_768_A_12_squad2_covid_qna Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid.bert.uncased_6l_128d_a2a_128d bert_qa_bert_uncased_L_6_H_128_A_2_squad2_covid_qna Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid.distil_bert.base_uncased distilbert_qa_base_uncased_squad2_covid_qa_deepset Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2_covid.electra.base electra_qa_base_squad2_covid_deepset Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid.roberta.base.by_armageddon roberta_qa_roberta_base_squad2_covid_qa_deepset Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2_covid.roberta.base.by_deepset roberta_qa_roberta_base_squad2_covid Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2_covid.roberta.large roberta_qa_roberta_large_squad2_covid_qa_deepset Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2_covid_cord19.bert.uncased_10l_512d_a8a_512d bert_qa_bert_uncased_L_10_H_512_A_8_cord19_200616_squad2_covid_qna Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid_cord19.bert.uncased_4l_256d_a4a_256d bert_qa_bert_uncased_L_4_H_256_A_4_cord19_200616_squad2_covid_qna Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid_cord19.bert.uncased_4l_512d_a8a_512d bert_qa_bert_uncased_L_4_H_512_A_8_cord19_200616_squad2_covid_qna Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid_cord19.bert.uncased_4l_768d_a12a_768d bert_qa_bert_uncased_L_4_H_768_A_12_cord19_200616_squad2_covid_qna Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_pubmed.bert.v2 bert_qa_pubmed_bert_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_pubmed.biobert.v2 bert_qa_biobert_v1.1_pubmed_squad_v2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_pubmed.sapbert bert_qa_sapbert_from_pubmedbert_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.synqa.electra.large electra_qa_large_synqa Question Answering English BertForQuestionAnswering
en.answer_question.synqa.roberta.large.by_mbartolo roberta_qa_roberta_large_synqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.synqa_ext.roberta.large.by_mbartolo roberta_qa_roberta_large_synqa_ext Question Answering English RoBertaForQuestionAnswering
en.answer_question.tquad.bert.xtremedistiled_uncased bert_qa_xtremedistil_l6_h256_uncased_TQUAD_finetuned_lr_2e_05_epochs_9 Question Answering English BertForQuestionAnswering
en.answer_question.trial.bert.by_sunitha bert_qa_Trial_3_Results Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid.bert.uncased_2l_512d_a8a_512d bert_qa_bert_uncased_L_2_H_512_A_8_squad2_covid_qna Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid.bert.uncased_10l_512d_a8a_512d bert_qa_bert_uncased_L_10_H_512_A_8_squad2_covid_qna Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid.bert.large_uncased bert_qa_bert_large_uncased_squad2_covid_qa_deepset Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid.bert.base_uncased bert_qa_bert_base_uncased_squad2_covid_qa_deepset Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.distilled_base xlm_roberta_qa_xlm_roberta_base_squad2_distilled Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.large xlm_roberta_qa_xlm_roberta_large_squad2 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2_bioasq8b.electra.base electra_qa_BioM_Base_SQuAD2_BioASQ8B Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_bioasq8b.electra.large electra_qa_BioM_Large_SQuAD2_BioASQ8B Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_chaii.xlm_roberta.distilled_base xlm_roberta_qa_xlm_roberta_base_squad2_distilled_finetuned_chaii Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2_chaii.xlm_roberta.distilled_base_small xlm_roberta_qa_xlm_roberta_base_squad2_distilled_finetuned_chaii_small Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2_chemical.bert.uncased bert_qa_chemical_bert_uncased_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_conll.bert.large_uncased.by_andi611 bert_qa_bert_large_uncased_whole_word_masking_squad2_with_ner_conll2003_with_neg_with_repeat Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_conll.bert.large_uncased_pistherea.by_andi611 bert_qa_bert_large_uncased_whole_word_masking_squad2_with_ner_Pistherea_conll2003_with_neg_with_repeat Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_conll.bert.large_uncased_pwhatisthe.by_andi611 bert_qa_bert_large_uncased_whole_word_masking_squad2_with_ner_Pwhatisthe_conll2003_with_neg_with_repeat Question Answering English BertForQuestionAnswering
en.answer_question.trivia.albert.xxl albert_qa_xxlarge_tweetqa Question Answering English AlbertForQuestionAnswering
en.answer_question.squadv2_conll.distil_bert.base_uncased.by_andi611 distilbert_qa_base_uncased_squad2_with_ner Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2_conll.distil_bert.base_uncased_with_neg_with_multi.by_andi611 distilbert_qa_base_uncased_squad2_with_ner_with_neg_with_multi Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2_conll.distil_bert.base_uncased_with_neg_with_multi_with_repeat.by_andi611 distilbert_qa_base_uncased_squad2_with_ner_with_neg_with_multi_with_repeat Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2_conll.distil_bert.base_uncased_with_neg_with_repeat.by_andi611 distilbert_qa_base_uncased_squad2_with_ner_with_neg_with_repeat Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2_cord19.bert.small bert_qa_bert_small_cord19_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_cord19.bert.uncased_10l_512d_a8a_512d bert_qa_bert_uncased_L_10_H_512_A_8_cord19_200616_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_cord19.bert.uncased_2l_512d_a8a_512d bert_qa_bert_uncased_L_2_H_512_A_8_cord19_200616_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_cord19.bert.uncased_4l_256d_a4a_256d bert_qa_bert_uncased_L_4_H_256_A_4_cord19_200616_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_cord19.bert.uncased_4l_512d_a8a_512d bert_qa_bert_uncased_L_4_H_512_A_8_cord19_200616_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_cord19.bert.uncased_4l_768d_a12a_768d bert_qa_bert_uncased_L_4_H_768_A_12_cord19_200616_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2_covid.albert.xxl_v2 albert_qa_xxlarge_v2_squad2_covid_deepset Question Answering English AlbertForQuestionAnswering
en.answer_question.squadv2_conll.distil_bert.base_uncased_with_neg.by_andi611 distilbert_qa_base_uncased_squad2_with_ner_with_neg Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_v2 xlm_roberta_qa_squadv2_xlm_roberta_base Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.trivia.bert.base_1024d bert_qa_bert_base_1024_full_trivia_copied_embeddings Question Answering English BertForQuestionAnswering
en.answer_question.trivia.bert.base_4096.by_MrAnderson bert_qa_bert_base_4096_full_trivia_copied_embeddings Question Answering English BertForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265898 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265898 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265899 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265899 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265900 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265900 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265901 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265901 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265902 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265902 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265903 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265903 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265904 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265904 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265905 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265905 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265906 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265906 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265907 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265907 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265908 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265908 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265909 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265909 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265910 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265910 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265911 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265911 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.fr_tuned.by_Gantenbein roberta_qa_ADDI_FR_XLM_R Question Answering English RoBertaForQuestionAnswering
en.answer_question.xlmr_roberta xlm_roberta_qa_XLMr_ENIS_QA_IsQ_EnA Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xquad.bert.multilingual_base bert_qa_bert_base_multilingual_xquad Question Answering English BertForQuestionAnswering
en.answer_question.xquad.xlm_roberta.base xlm_roberta_qa_xlm_roberta_base_xquad Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xquad.xlm_roberta.large xlm_roberta_qa_xlm_roberta_large_xquad Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xquad_chaii.bert.cased bert_qa_bert_multi_cased_finedtuned_xquad_chaii Question Answering English BertForQuestionAnswering
en.answer_question.xquad_squad.bert.cased bert_qa_bert_multi_cased_finetuned_xquadv1_finetuned_squad_colab Question Answering English BertForQuestionAnswering
en.answer_question.xlm_roberta.fine_tune_24465520_26265897 xlm_roberta_qa_autonlp_more_fine_tune_24465520_26265897 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.by_ncthuan xlm_roberta_qa_xlm_l_uetqa Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.by_laifuchicago xlm_roberta_qa_farm2tran Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.by_jeew xlm_roberta_qa_xlm_roberta_ckpt_95000 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.trivia.bert.base_512d bert_qa_bert_base_512_full_trivia Question Answering English BertForQuestionAnswering
en.answer_question.trivia.bert.by_Danastos bert_qa_triviaqa_bert_el_Danastos Question Answering English BertForQuestionAnswering
en.answer_question.trivia.bert.by_Kutay bert_qa_fine_tuned_tweetqa_aip Question Answering English BertForQuestionAnswering
en.answer_question.trivia.distil_bert.base_uncased distilbert_qa_base_uncased_finetuned_triviaqa Question Answering English DistilBertForQuestionAnswering
en.answer_question.trivia.longformer.large longformer_qa_large_4096_finetuned_triviaqa Question Answering English LongformerForQuestionAnswering
en.answer_question.trivia.roberta roberta_qa_roberta_fine_tuned_tweet_sentiment_extractor Question Answering English RoBertaForQuestionAnswering
en.answer_question.trivia.roberta.base roberta_qa_roberta_base_tweetqa_model Question Answering English RoBertaForQuestionAnswering
en.answer_question.trivia.roberta.large roberta_qa_roberta_large_tweetqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.trivia.xlmr_roberta.large xlm_roberta_qa_xlmroberta_large_tweetqa Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.tydiqa.bert bert_qa_bert_all Question Answering English BertForQuestionAnswering
en.answer_question.trivia.bert.base_2048.by_MrAnderson bert_qa_bert_base_2048_full_trivia_copied_embeddings Question Answering English BertForQuestionAnswering
en.answer_question.tydiqa.bert.multilingual bert_qa_Part_2_BERT_Multilingual_Dutch_Model_E1 Question Answering English BertForQuestionAnswering
en.answer_question.tydiqa.multi_lingual_bert bert_qa_Part_2_mBERT_Model_E2 Question Answering English BertForQuestionAnswering
en.answer_question.tydiqa.roberta roberta_qa_roberta_tydiqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.tydiqa.xlm_roberta.3lang xlm_roberta_qa_xlm_3lang Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.tydiqa.xlm_roberta.by_horsbug98 xlm_roberta_qa_Part_1_XLM_Model_E1 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.tydiqa.xlm_roberta.by_krinal214 xlm_roberta_qa_xlm_all Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.tydiqa.xlm_roberta.v2.by_horsbug98 xlm_roberta_qa_Part_2_XLM_Model_E1 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.base xlm_roberta_qa_xlm_roberta_base_finetune_qa Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.by_Dongjae xlm_roberta_qa_mrc2reader Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.by_Srini99 xlm_roberta_qa_TQA Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.xlm_roberta.by_anukaver xlm_roberta_qa_xlm_roberta_est_qa Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.tydiqa.distil_bert distilbert_qa_multi_finetuned_for_xqua_on_tydiqa Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2.bert.large_uncased.by_Salesforce bert_qa_qaconv_bert_large_uncased_whole_word_masking_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_24465525.by_teacookies xlm_roberta_qa_autonlp_roberta_base_squad2_24465525 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_24465523.by_teacookies xlm_roberta_qa_autonlp_roberta_base_squad2_24465523 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.distil_bert.base_uncased.by_andi611 distilbert_qa_base_uncased_squad2_with_ner_mit_restaurant_with_neg_with_repeat Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2.distil_bert.base_uncased.by_anurag0077 distilbert_qa_anurag0077_base_uncased_finetuned_squad2 Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2.distil_bert.base_uncased.by_mvonwyl distilbert_qa_mvonwyl_base_uncased_finetuned_squad2 Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2.distil_bert.base_uncased.by_tabo distilbert_qa_tabo_base_uncased_finetuned_squad2 Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2.distil_bert.base_uncased.by_twmkn9 distilbert_qa_base_uncased_squad2 Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2.distil_bert.by_threem distilbert_qa_mysquadv2_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2.distil_bert.v2.by_threem distilbert_qa_mysquadv2_8Jan22_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2.electra.base.by_PremalMatalia electra_qa_base_best_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.electra.base.by_navteca electra_qa_base_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.electra.base.by_sultan electra_qa_BioM_Base_SQuAD2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.electra.base_v2 electra_qa_base_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.electra.large.by_sultan electra_qa_BioM_Large_SQuAD2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.electra.large.by_superspray electra_qa_large_discriminator_squad2_custom_dataset Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.electra.large_512d electra_qa_large_discriminator_squad2_512 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.electra.small_v2 electra_qa_small_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.longformer.base longformer_base_base_qa_squad2 Question Answering English LongformerForQuestionAnswering
en.answer_question.squadv2.longformer.base_v2 longformer_qa_base_4096_finetuned_squadv2 Question Answering English LongformerForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_21iridescent roberta_qa_RoBERTa_base_finetuned_squad2_lwt Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_AnonymousSub roberta_qa_roberta_base_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_PremalMatalia roberta_qa_roberta_base_best_squad2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_Shappey roberta_qa_roberta_base_QnA_squad2_trained Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.distil_bert.base_cased distilbert_base_cased_qa_squad2 Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2.distil_bert.base distilbert_qa_base_squad2_custom_dataset Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2.deberta deberta_v3_xsmall_qa_squad2 Question Answering English DeBertaForQuestionAnswering
en.answer_question.squadv2.biobert.cased.by_ptnv-s bert_qa_biobert_squad2_cased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.large_uncased.by_deepset bert_qa_bert_large_uncased_whole_word_masking_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.large_uncased_v2.by_madlag bert_qa_bert_large_uncased_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.large_uncased_v2_x2.15_f83.2_d25_hybrid.by_madlag bert_qa_bert_large_uncased_wwm_squadv2_x2.15_f83.2_d25_hybrid_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.large_uncased_v2_x2.63_f82.6_d16_hybrid.by_madlag bert_qa_bert_large_uncased_wwm_squadv2_x2.63_f82.6_d16_hybrid_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.large_uncased_whole_word_masking_v2.by_madlag bert_qa_bert_large_uncased_whole_word_masking_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.medium_v2 bert_qa_bert_medium_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.mini_lm_base_uncased bert_qa_minilm_uncased_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.small.by_mrm8488 bert_qa_bert_small_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.small_v2.by_mrm8488 bert_qa_bert_small_2_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.tiny_.by_mrm8488 bert_qa_bert_tiny_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_Teepika roberta_qa_roberta_base_squad2_finetuned_selqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_fadhilarkan distilbert_qa_fadhilarkan_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squadv2.bert.tiny_v3.by_mrm8488 bert_qa_bert_tiny_3_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.tiny_v4.by_mrm8488 bert_qa_bert_tiny_4_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.tiny_v5.by_mrm8488 bert_qa_bert_tiny_5_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.uncased_10l_512d_a8a_512d bert_qa_bert_uncased_L_10_H_512_A_8_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.uncased_2l_512d_a8a_512d bert_qa_bert_uncased_L_2_H_512_A_8_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.uncased_4l_256d_a4a_256d bert_qa_bert_uncased_L_4_H_256_A_4_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.uncased_4l_512d_a8a_512d bert_qa_bert_uncased_L_4_H_512_A_8_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.uncased_4l_768d_a12a_768d bert_qa_bert_uncased_L_4_H_768_A_12_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.uncased_6l_128d_a2a_128d bert_qa_bert_uncased_L_6_H_128_A_2_squad2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.biobert.cased.by_clagator bert_qa_biobert_squad2_cased Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.bert.tiny_v2.by_mrm8488 bert_qa_bert_tiny_2_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_24465524.by_teacookies xlm_roberta_qa_autonlp_roberta_base_squad2_24465524 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_avioo1 roberta_qa_avioo1_roberta_base_squad2_finetuned_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_deepset roberta_base_qa_squad2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.distilled_base_128d_32d_v2 roberta_qa_distilrobert_base_squadv2_328seq_128stride_test Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.distilled_base_v2 roberta_qa_distilroberta_base_squad_v2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.emanuals.by_AnonymousSub roberta_qa_EManuals_RoBERTa_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.large.by_Salesforce roberta_qa_qaconv_roberta_large_squad2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.large.by_deepset roberta_qa_roberta_large_squad2_hp Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.large.by_navteca roberta_qa_roberta_large_squad2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.large.by_phiyodr roberta_qa_roberta_large_finetuned_squad2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.tiny.by_deepset roberta_qa_tinyroberta_squad2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.tiny.v2.by_deepset roberta_qa_tinyroberta_squad2_step1 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.scibert.uncased_v2 bert_qa_scibert_scivocab_uncased_squad_v2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.span_bert.v2 bert_qa_spanbert_finetuned_squadv2 Question Answering English BertForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base.by_deepset xlm_roberta_base_qa_squad2 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_24465514.by_teacookies xlm_roberta_qa_autonlp_roberta_base_squad2_24465514 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_24465515.by_teacookies xlm_roberta_qa_autonlp_roberta_base_squad2_24465515 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_24465516.by_teacookies xlm_roberta_qa_autonlp_roberta_base_squad2_24465516 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_24465517.by_teacookies xlm_roberta_qa_autonlp_roberta_base_squad2_24465517 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_24465518.by_teacookies xlm_roberta_qa_autonlp_roberta_base_squad2_24465518 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_24465519.by_teacookies xlm_roberta_qa_autonlp_roberta_base_squad2_24465519 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_24465520.by_teacookies xlm_roberta_qa_autonlp_roberta_base_squad2_24465520 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_24465521.by_teacookies xlm_roberta_qa_autonlp_roberta_base_squad2_24465521 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.xlm_roberta.base_24465522.by_teacookies xlm_roberta_qa_autonlp_roberta_base_squad2_24465522 Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.distilled_base.by_twmkn9 roberta_qa_distilroberta_base_squad2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.distilled_base.by_deepset roberta_qa_roberta_base_squad2_distilled Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.distilled_base.by_21iridescent roberta_qa_distilroberta_base_finetuned_squad2_lwt Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.declutr.by_AnonymousSub roberta_qa_declutr_model_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_mvonwyl roberta_qa_roberta_base_finetuned_squad2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_navteca roberta_qa_navteca_roberta_base_squad2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_nlpconnect roberta_qa_roberta_base_squad2_nq Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_prk roberta_qa_prk_roberta_base_squad2_finetuned_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_shahrukhx01 roberta_qa_roberta_base_squad2_boolq_baseline Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_sumba roberta_qa_sumba_roberta_base_squad2_finetuned_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_ydshieh roberta_qa_ydshieh_roberta_base_squad2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_rule_based_hier_quadruplet_0.1_epochs_1_shard_1.by_AnonymousSub roberta_qa_rule_based_roberta_hier_quadruplet_0.1_epochs_1_shard_1_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_rule_based_hier_quadruplet_epochs_1_shard_1.by_AnonymousSub roberta_qa_rule_based_roberta_hier_quadruplet_epochs_1_shard_1_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_rule_based_hier_triplet_0.1_epochs_1_shard_1.by_AnonymousSub roberta_qa_rule_based_roberta_hier_triplet_0.1_epochs_1_shard_1_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base.by_deepakvk roberta_qa_deepakvk_roberta_base_squad2_finetuned_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_rule_based_hier_triplet_epochs_1_shard_1.by_AnonymousSub roberta_qa_rule_based_roberta_hier_triplet_epochs_1_shard_1_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_rule_based_only_classfn_twostage_epochs_1_shard_1.by_AnonymousSub roberta_qa_rule_based_roberta_only_classfn_twostage_epochs_1_shard_1_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_rule_based_quadruplet_epochs_1_shard_1.by_AnonymousSub roberta_qa_rule_based_roberta_bert_quadruplet_epochs_1_shard_1_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_rule_based_twostage_quadruplet_epochs_1_shard_1.by_AnonymousSub roberta_qa_rule_based_roberta_twostage_quadruplet_epochs_1_shard_1_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_rule_based_twostagequadruplet_hier_epochs_1_shard_1.by_AnonymousSub roberta_qa_rule_based_roberta_twostagequadruplet_hier_epochs_1_shard_1_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_rule_based_twostagetriplet_epochs_1_shard_1.by_AnonymousSub roberta_qa_rule_based_roberta_twostagetriplet_epochs_1_shard_1_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_rule_based_twostagetriplet_hier_epochs_1_shard_1.by_AnonymousSub roberta_qa_rule_based_roberta_twostagetriplet_hier_epochs_1_shard_1_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_ruletriplet_epochs_1_shard_1.by_AnonymousSub roberta_qa_rule_based_roberta_bert_triplet_epochs_1_shard_1_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_v2.by_AyushPJ roberta_qa_ai_club_inductions_21_nlp_roBERTa_base_squad_v2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_v2.by_mrm8488 roberta_qa_roberta_base_1B_1_finetuned_squadv2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.cline.by_AnonymousSub roberta_qa_cline_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squadv2.roberta.base_rule_based_only_classfn_epochs_1_shard_1.by_AnonymousSub roberta_qa_rule_based_roberta_only_classfn_epochs_1_shard_1_squad2.0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_en distilbert_qa_en_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.electra.large.by_mrm8488 electra_qa_large_finetuned_squadv1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_deepakvk distilbert_qa_base_uncased_distilled_squad_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.single_label_n_max.by_mcurmei distilbert_qa_single_label_N_max Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.single_label_n_max_long_training.by_mcurmei distilbert_qa_single_label_N_max_long_training Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.unique_n_max.by_mcurmei distilbert_qa_unique_N_max Question Answering English DistilBertForQuestionAnswering
en.answer_question.electra.by_Andranik electra_qa_TestQA2 Question Answering English BertForQuestionAnswering
en.answer_question.electra.by_carlosserquen electra_qa_elctrafp Question Answering English BertForQuestionAnswering
en.answer_question.electra.by_rowan1224 electra_qa_slp Question Answering English BertForQuestionAnswering
en.answer_question.electra.finetuning_1 electra_qa_DSPFirst_Finetuning_1 Question Answering English BertForQuestionAnswering
en.answer_question.electra.finetuning_2 electra_qa_DSPFirst_Finetuning_2 Question Answering English BertForQuestionAnswering
en.answer_question.electra.finetuning_3 electra_qa_DSPFirst_Finetuning_3 Question Answering English BertForQuestionAnswering
en.answer_question.electra.finetuning_4 electra_qa_DSPFirst_Finetuning_4 Question Answering English BertForQuestionAnswering
en.answer_question.electra.finetuning_5 electra_qa_DSPFirst_Finetuning_5 Question Answering English BertForQuestionAnswering
en.answer_question.klue.bert bert_qa_Klue_CommonSense_model Question Answering English BertForQuestionAnswering
en.answer_question.klue.bert.multilingual_base_cased bert_qa_bert_base_multilingual_cased_finetuned_klue Question Answering English BertForQuestionAnswering
en.answer_question.klue.xlm_roberta.base xlm_roberta_qa_klue_mrc_roberta_base Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_emre distilbert_qa_emre_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.korquad.bert.multilingual_base_cased.by_sangrimlee bert_qa_bert_base_multilingual_cased_korquad Question Answering English BertForQuestionAnswering
en.answer_question.korquad.xlm_roberta.large xlm_roberta_qa_xlm_roberta_large_korquad_mask Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.longformer.by_Nomi97 longformer_qa_Chatbot Question Answering English LongformerForQuestionAnswering
en.answer_question.longformer.by_manishiitg longformer_qa_recruit Question Answering English LongformerForQuestionAnswering
en.answer_question.longformer.by_ponmari longformer_qa_ponmari Question Answering English LongformerForQuestionAnswering
en.answer_question.longformer.large longformer_qa_recruit_large Question Answering English LongformerForQuestionAnswering
en.answer_question.distil_bert.log_parser_winlogbeat.by_Slavka distilbert_qa_distil_bert_finetuned_log_parser_winlogbeat Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.log_parser.by_Slavka distilbert_qa_distil_bert_finetuned_log_parser_1 Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.flat_n_max.by_mcurmei distilbert_qa_flat_N_max Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.custom5.by_aszidon distilbert_qa_custom5 Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.base_cased.by_adamlin distilbert_qa_base_cased_sgd_qa_step5000 Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.base_config1.by_nlpunibo distilbert_qa_base_config1 Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.base_config2.by_nlpunibo distilbert_qa_base_config2 Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.base_config3.by_nlpunibo distilbert_qa_base_config3 Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.base_uncased.by_T-qualizer distilbert_qa_base_uncased_finetuned_advers Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.base_uncased.by_charlieoneill distilbert_qa_base_uncased_gradient_clinic Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.base_uncased.by_datarpit distilbert_qa_base_uncased_finetuned_natural_questions Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.base_uncased.by_machine2049 distilbert_qa_base_uncased_finetuned_duorc_ Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.base_uncased.by_tiennvcs distilbert_qa_base_uncased_finetuned_infovqa Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.by_Ifenna distilbert_qa_dbert_3epoch Question Answering English DistilBertForQuestionAnswering
en.answer_question.longformer.v2 longformer_qa_recruit_v2 Question Answering English LongformerForQuestionAnswering
en.answer_question.distil_bert.by_LucasS distilbert_qa_distilBertABSA Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.by_Sounak distilbert_qa_finetuned Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.by_ajaypyatha distilbert_qa_sdsqna Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.by_alinemati distilbert_qa_BERT Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.by_keras-io distilbert_qa_transformers_qa Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.by_minhdang241 distilbert_qa_robustqa_tapt Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.by_pakupoko distilbert_qa_bizlin_distil_model Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.by_poom-sci distilbert_qa_qa Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.custom.by_aszidon distilbert_qa_custom Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.custom3.by_aszidon distilbert_qa_custom3 Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.custom4.by_aszidon distilbert_qa_custom4 Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.by_Sarmad distilbert_qa_projectmodel_bert Question Answering English DistilBertForQuestionAnswering
en.answer_question.distil_bert.base_cased.by_Slavka distilbert_qa_bert_base_cased_finetuned_log_parser_winlogbeat Question Answering English DistilBertForQuestionAnswering
en.answer_question.mitmovie_squad.roberta.by_thatdramebaazguy roberta_qa_movie_roberta_MITmovie_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.mlqa.bert.base_uncased bert_qa_bert_base_spanish_wwm_uncased_finetuned_qa_mlqa Question Answering English BertForQuestionAnswering
en.answer_question.news.roberta.qa_ft.by_AnonymousSub roberta_qa_news_pretrain_roberta_FT_newsqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.news.roberta.qa_ft_new.by_AnonymousSub roberta_qa_news_pretrain_roberta_FT_new_newsqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.news.roberta.qa_roberta_ft_new_newsqa.by_AnonymousSub roberta_qa_roberta_FT_new_newsqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.news.roberta.qa_roberta_ft_newsqa.by_AnonymousSub roberta_qa_roberta_FT_newsqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.output_files.bert.by_sunitha bert_qa_output_files Question Answering English BertForQuestionAnswering
en.answer_question.pubmed.bert.base_uncased.by_Shushant bert_qa_Shushant_BiomedNLP_PubMedBERT_base_uncased_abstract_fulltext_ContaminationQAmodel_PubmedBERT Question Answering English BertForQuestionAnswering
en.answer_question.roberta.756523213.by_AlirezaBaneshi roberta_qa_autotrain_test2_756523213 Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.756523214.by_AlirezaBaneshi roberta_qa_autotrain_test2_756523214 Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.augmented roberta_qa_roberta_unaugmentedv3 Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.base.by_123tarunanand roberta_qa_roberta_base_finetuned Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.base.by_eAsyle roberta_qa_roberta_base_custom_QA Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.base.by_emr-se-miniproject roberta_qa_roberta_base_emr Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.base.by_nlpconnect roberta_qa_dpr_nq_reader_roberta_base Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.base.by_rsvp-ai roberta_qa_bertserini_roberta_base Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.base_v2 roberta_qa_dpr_nq_reader_roberta_base_v2 Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_AmazonScience roberta_qa_qanlu Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_Andranik roberta_qa_TestQaV1 Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_AyushPJ roberta_qa_ai_club_inductions_21_nlp_roBERTa Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_Beri roberta_qa_legal_qa Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_CNT-UPenn roberta_qa_RoBERTa_for_seizureFrequency_QA Question Answering English RoBertaForQuestionAnswering
en.answer_question.news.roberta.qa_fpdm_triplet_roberta_ft_newsqa.by_AnonymousSub roberta_qa_fpdm_triplet_roberta_FT_newsqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.news.roberta.qa_fpdm_triplet_roberta_ft_new_newsqa.by_AnonymousSub roberta_qa_fpdm_triplet_roberta_FT_new_newsqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.news.roberta.qa_fpdm_roberta_ft_newsqa.by_AnonymousSub roberta_qa_fpdm_roberta_FT_newsqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.news.roberta.qa_fpdm_hier_roberta_ft_newsqa.by_AnonymousSub roberta_qa_fpdm_hier_roberta_FT_newsqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.movie_squad.roberta.base roberta_qa_roberta_base_MITmovie_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.movie_squad.roberta.by_thatdramebaazguy roberta_qa_movie_roberta_squad Question Answering English RoBertaForQuestionAnswering
en.answer_question.movie_squadv2.bert.large_uncased bert_qa_bert_large_uncased_whole_word_masking_squad2_with_ner_mit_movie_with_neg_with_repeat Question Answering English BertForQuestionAnswering
en.answer_question.multi_lingual_bert.by_horsbug98 bert_qa_Part_1_mBERT_Model_E2 Question Answering English BertForQuestionAnswering
en.answer_question.multi_lingual_bert.by_krinal214 bert_qa_mBERT_all_ty_SQen_SQ20_1 Question Answering English BertForQuestionAnswering
en.answer_question.news.bert.base_uncased.by_mirbostani bert_qa_bert_base_uncased_finetuned_newsqa Question Answering English BertForQuestionAnswering
en.answer_question.news.bert.base_uncased.by_tli8hf bert_qa_unqover_bert_base_uncased_newsqa Question Answering English BertForQuestionAnswering
en.answer_question.news.bert.by_AnonymousSub bert_qa_news_pretrain_bert_FT_newsqa Question Answering English BertForQuestionAnswering
en.answer_question.news.bert.by_Danastos bert_qa_newsqa_bert_el_Danastos Question Answering English BertForQuestionAnswering
en.answer_question.news.bert.fpdm_ft.by_AnonymousSub bert_qa_fpdm_bert_FT_newsqa Question Answering English BertForQuestionAnswering
en.answer_question.mlqa.bert.base_cased bert_qa_bert_base_spanish_wwm_cased_finetuned_qa_mlqa Question Answering English BertForQuestionAnswering
en.answer_question.news.bert.fpdm_ft_new.by_AnonymousSub bert_qa_fpdm_bert_FT_new_newsqa Question Answering English BertForQuestionAnswering
en.answer_question.news.bert.fpdm_hier_ft_by_AnonymousSub bert_qa_fpdm_hier_bert_FT_new_newsqa Question Answering English BertForQuestionAnswering
en.answer_question.news.bert.ft.by_AnonymousSub bert_qa_bert_FT_newsqa Question Answering English BertForQuestionAnswering
en.answer_question.news.bert.ft_new.by_AnonymousSub bert_qa_bert_FT_new_newsqa Question Answering English BertForQuestionAnswering
en.answer_question.news.bert.new.by_AnonymousSub bert_qa_news_pretrain_bert_FT_new_newsqa Question Answering English BertForQuestionAnswering
en.answer_question.news.bert.qa_fpdm_triplet_ft.by_AnonymousSub bert_qa_fpdm_triplet_bert_FT_newsqa Question Answering English BertForQuestionAnswering
en.answer_question.news.bert.qa_fpdm_triplet_ft_new.by_AnonymousSub bert_qa_fpdm_triplet_bert_FT_new_newsqa Question Answering English BertForQuestionAnswering
en.answer_question.news.distil_bert.base_uncased distilbert_qa_unqover_base_uncased_newsqa Question Answering English DistilBertForQuestionAnswering
en.answer_question.news.roberta.base roberta_qa_unqover_roberta_base_newsqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.news.roberta.large roberta_qa_unqover_roberta_large_newsqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.news.roberta.qa_fpdm_hier_roberta_ft_new_newsqa.by_AnonymousSub roberta_qa_fpdm_hier_roberta_FT_new_newsqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.news.bert.fpdm_hier_ft.by_AnonymousSub bert_qa_fpdm_hier_bert_FT_newsqa Question Answering English BertForQuestionAnswering
en.answer_question.roberta.by_Ching roberta_qa_negation_detector Question Answering English RoBertaForQuestionAnswering
en.answer_question.distil_bert.base.by_minhdang241 distilbert_qa_robustqa_baseline_01 Question Answering English DistilBertForQuestionAnswering
en.answer_question.cuad_gam.roberta.base.by_Gam roberta_qa_roberta_base_finetuned_cuad_gam Question Answering English RoBertaForQuestionAnswering
en.answer_question.bert.by_SanayCo bert_qa_model_output Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_aymanm419 bert_qa_araSpeedest Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_ericRosello bert_qa_results Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_internetoftim bert_qa_demo Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_jackh1995 bert_qa_bert_finetuned_jackh1995 Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_krinal214 bert_qa_bert_all_translated Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_manav bert_qa_causal_qa Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_mezes bert_qa_eauction_section_parsing_from_pretrained Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_motiondew bert_qa_bert_finetuned_lr2_e5_b16_ep2 Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_mrm8488 bert_qa_ManuERT_for_xqua Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_nlpunibo bert_qa_bert Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_nvkha bert_qa_bert_qa_vi_nvkha Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_piEsposito bert_qa_braquad_bert_qna Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_voidful bert_qa_question_answering_zh_voidful Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_z-uo bert_qa_bert_qasper Question Answering English BertForQuestionAnswering
en.answer_question.bert.distilled_base_uncased bert_qa_distilbert_base_uncased_finetuned_custom Question Answering English BertForQuestionAnswering
en.answer_question.bert.docvqa.base_uncased.by_tiennvcs bert_qa_bert_base_uncased_finetuned_docvqa Question Answering English BertForQuestionAnswering
en.answer_question.bert.infovqa.base_uncased.by_tiennvcs bert_qa_bert_base_uncased_finetuned_infovqa Question Answering English BertForQuestionAnswering
en.answer_question.bert.large.by_Sounak bert_qa_bert_large_finetuned Question Answering English BertForQuestionAnswering
en.answer_question.bert.large.by_atharvamundada99 bert_qa_bert_large_question_answering_finetuned_legal Question Answering English BertForQuestionAnswering
en.answer_question.bert.large.by_ricardo-filho bert_qa_bert_large_faquad Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_Rocketknight1 bert_qa_bert_finetuned_qa Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_LenaSchmidt bert_qa_no_need_to_name_this Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_HankyStyle bert_qa_Multi_ling_BERT Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_ForutanRad bert_qa_bert_fa_QA_v1 Question Answering English BertForQuestionAnswering
en.answer_qu estion.mqa_cls.bert.by_xraychen bert_qa_mqa_cls Question Answering English BertForQuestionAnswering
en.answer_question.albert.by_AyushPJ albert_qa_ai_club_inductions_21_nlp Question Answering English AlbertForQuestionAnswering
en.answer_question.albert.by_SalmanMo albert_qa_QA_1e Question Answering English AlbertForQuestionAnswering
en.answer_question.albert.by_nlpunibo albert_qa_nlpunibo Question Answering English AlbertForQuestionAnswering
en.answer_question.albert.by_rowan1224 albert_qa_slp Question Answering English AlbertForQuestionAnswering
en.answer_question.albert.by_saburbutt albert_qa_generic Question Answering English AlbertForQuestionAnswering
en.answer_question.albert.xl albert_qa_xlarge_finetuned Question Answering English AlbertForQuestionAnswering
en.answer_question.bert.32d bert_qa_bert_set_date_1_lr_2e_5_bs_32_ep_4 Question Answering English BertForQuestionAnswering
en.answer_question.bert.augmented bert_qa_augmented Question Answering English BertForQuestionAnswering
en.answer_question.bert.base.by_peggyhuang bert_qa_finetune_bert_base_v1 Question Answering English BertForQuestionAnswering
en.answer_question.bert.large_cased bert_qa_muril_large_cased_hita_qa Question Answering English BertForQuestionAnswering
en.answer_question.bert.base.by_ricardo-filho bert_qa_bert_base_faquad Question Answering English BertForQuestionAnswering
en.answer_question.bert.base_cased.by_CenIA bert_qa_bert_base_spanish_wwm_cased_finetuned_qa_tar Question Answering English BertForQuestionAnswering
en.answer_question.bert.base_cased.by_husnu bert_qa_bert_base_turkish_cased_finetuned_lr_2e_05_epochs_3 Question Answering English BertForQuestionAnswering
en.answer_question.bert.base_cased.by_nntadotzip bert_qa_bert_base_cased_IUChatbot_ontologyDts Question Answering English BertForQuestionAnswering
en.answer_question.bert.base_uncased.by_CenIA bert_qa_bert_base_spanish_wwm_uncased_finetuned_qa_tar Question Answering English BertForQuestionAnswering
en.answer_question.bert.base_uncased.by_machine2049 bert_qa_bert_base_uncased_finetuned_duorc_bert Question Answering English BertForQuestionAnswering
en.answer_question.bert.base_uncased.by_peggyhuang bert_qa_bert_base_uncased_coqa Question Answering English BertForQuestionAnswering
en.answer_question.bert.base_uncased.by_vanadhi bert_qa_bert_base_uncased_fiqa_flm_sq_flit Question Answering English BertForQuestionAnswering
en.answer_question.bert.base_v2 bert_qa_finetune_bert_base_v2 Question Answering English BertForQuestionAnswering
en.answer_question.bert.base_v3.by_peggyhuang bert_qa_finetune_bert_base_v3 Question Answering English BertForQuestionAnswering
en.answer_question.bert.by_Danastos bert_qa_nq_bert_el_Danastos Question Answering English BertForQuestionAnswering
en.answer_question.bert.base.by_xraychen bert_qa_mqa_baseline Question Answering English BertForQuestionAnswering
en.answer_question.distil_bert.base.by_leemii18 distilbert_qa_robustqa_baseline_02 Question Answering English DistilBertForQuestionAnswering
en.answer_question.bert.large_uncased bert_qa_bert_large_uncased_finetuned_docvqa Question Answering English BertForQuestionAnswering
en.answer_question.bert.multilingual_english_tuned_base_cased.by_bhavikardeshna bert_qa_multilingual_bert_base_cased_english Question Answering English BertForQuestionAnswering
en.answer_question.chaii.xlm_roberta.base.by_SauravMaheshkar xlm_roberta_qa_xlm_roberta_base_chaii Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.chaii.xlm_roberta.base.by_tyqiangz xlm_roberta_qa_xlm_roberta_base_finetuned_chaii Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.chaii.xlm_roberta.large.by_SauravMaheshkar xlm_roberta_qa_xlm_roberta_large_chaii Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.chaii.xlm_roberta.large_multi.by_SauravMaheshkar xlm_roberta_qa_xlm_multi_roberta_large_chaii Question Answering English XlmRoBertaForQuestionAnswering
en.answer_question.clinical.distil_bert distilbert_qa_BERT_ClinicalQA Question Answering English DistilBertForQuestionAnswering
en.answer_question.conll.distil_bert.base_uncased distilbert_qa_base_uncased_qa_with_ner Question Answering English DistilBertForQuestionAnswering
en.answer_question.cord19.bert.by_JAlexis bert_qa_Bertv1_fine Question Answering English BertForQuestionAnswering
en.answer_question.cord19.bert.small bert_qa_bert_small_cord19qa Question Answering English BertForQuestionAnswering
en.answer_question.cord19.prueba_bert.by_JAlexis bert_qa_PruebaBert Question Answering English BertForQuestionAnswering
en.answer_question.covid.distil_bert.a.by_rahulkuruvilla distilbert_qa_COVID_DistilBERTa Question Answering English DistilBertForQuestionAnswering
en.answer_question.covid.distil_bert.b.by_rahulkuruvilla distilbert_qa_COVID_DistilBERTb Question Answering English DistilBertForQuestionAnswering
en.answer_question.covid.distil_bert.c.by_rahulkuruvilla distilbert_qa_COVID_DistilBERTc Question Answering English DistilBertForQuestionAnswering
en.answer_question.covid.longformer longformer_qa_covid Question Answering English LongformerForQuestionAnswering
en.answer_question.covid_bert.a.by_rahulkuruvilla bert_qa_COVID_BERTa Question Answering English BertForQuestionAnswering
en.answer_question.covid_bert.b.by_rahulkuruvilla bert_qa_COVID_BERTb Question Answering English BertForQuestionAnswering
en.answer_question.covid_bert.c.by_rahulkuruvilla bert_qa_COVID_BERTc Question Answering English BertForQuestionAnswering
en.answer_question.cuad.roberta.base.by_Gam roberta_qa_roberta_base_finetuned_cuad Question Answering English RoBertaForQuestionAnswering
en.answer_question.cuad.roberta.base.by_Rakib roberta_qa_roberta_base_on_cuad Question Answering English RoBertaForQuestionAnswering
en.answer_question.cuad.roberta.base.by_akdeniz27 roberta_qa_akdeniz27_roberta_base_cuad Question Answering English RoBertaForQuestionAnswering
en.answer_question.cuad.roberta.base.by_marshmellow77 roberta_qa_marshmellow77_roberta_base_cuad Question Answering English RoBertaForQuestionAnswering
en.answer_question.cuad.roberta.large roberta_qa_roberta_large_cuad Question Answering English RoBertaForQuestionAnswering
en.answer_question.chaii.roberta.base roberta_qa_roberta_base_chaii Question Answering English RoBertaForQuestionAnswering
en.answer_question.chaii.electra.base electra_qa_base_chaii Question Answering English BertForQuestionAnswering
en.answer_question.chaii.distil_bert.base_uncased distilbert_qa_base_uncased_distilled_chaii Question Answering English DistilBertForQuestionAnswering
en.answer_question.chaii.distil_bert.base_cased distilbert_qa_base_cased_distilled_chaii Question Answering English DistilBertForQuestionAnswering
en.answer_question.bert.multilingual_german_tuned_base_cased.by_bhavikardeshna bert_qa_multilingual_bert_base_cased_german Question Answering English BertForQuestionAnswering
en.answer_question.bert.multilingual_hindi_tuned_base_cased.by_bhavikardeshna bert_qa_multilingual_bert_base_cased_hindi Question Answering English BertForQuestionAnswering
en.answer_question.bert.multilingual_spanish_tuned_base_cased.by_bhavikardeshna bert_qa_multilingual_bert_base_cased_spanish Question Answering English BertForQuestionAnswering
en.answer_question.bert.multilingual_vietnamese_tuned_base_cased.by_bhavikardeshna bert_qa_multilingual_bert_base_cased_vietnamese Question Answering English BertForQuestionAnswering
en.answer_question.bert.sim.by_xraychen bert_qa_mqa_sim Question Answering English BertForQuestionAnswering
en.answer_question.bert.unsupsim.by_xraychen bert_qa_mqa_unsupsim Question Answering English BertForQuestionAnswering
en.answer_question.bert.vi_infovqa.base_uncased.by_tiennvcs bert_qa_bert_base_uncased_finetuned_vi_infovqa Question Answering English BertForQuestionAnswering
en.answer_question.bert.xtremedistiled_uncased_lr_2e_05_epochs_3.by_husnu bert_qa_xtremedistil_l6_h256_uncased_finetuned_lr_2e_05_epochs_3 Question Answering English BertForQuestionAnswering
en.answer_question.bert.xtremedistiled_uncased_lr_2e_05_epochs_6.by_husnu bert_qa_xtremedistil_l6_h256_uncased_finetuned_lr_2e_05_epochs_6 Question Answering English BertForQuestionAnswering
en.answer_question.bert.zero_shot.by_fractalego bert_qa_fewrel_zero_shot Question Answering English BertForQuestionAnswering
en.answer_question.bert.multilingual_arabic_tuned_base_cased.by_bhavikardeshna bert_qa_multilingual_bert_base_cased_arabic Question Answering English BertForQuestionAnswering
en.answer_question.bert.zero_shot.by_krinal214 bert_qa_zero_shot Question Answering English BertForQuestionAnswering
en.answer_question.bio_medical.bert.base bert_qa_biomedical_slot_filling_reader_base Question Answering English BertForQuestionAnswering
en.answer_question.bio_medical.bert.large bert_qa_biomedical_slot_filling_reader_large Question Answering English BertForQuestionAnswering
en.answer_question.biobert bert_qa_biobert_bioasq Question Answering English BertForQuestionAnswering
en.answer_question.chaii.bert.base_cased bert_qa_bert_base_cased_chaii Question Answering English BertForQuestionAnswering
en.answer_question.chaii.bert.cased bert_qa_bert_multi_cased_finetuned_chaii Question Answering English BertForQuestionAnswering
en.answer_question.chaii.bert.large_uncased_uncased_whole_word_masking.by_SauravMaheshkar bert_qa_bert_large_uncased_whole_word_masking_chaii Question Answering English BertForQuestionAnswering
en.answer_question.chaii.bert.large_uncased_uncased_whole_word_masking_finetuned.by_SauravMaheshkar bert_qa_bert_large_uncased_whole_word_masking_finetuned_chaii Question Answering English BertForQuestionAnswering
en.answer_question.chaii.bert.multilingual_base_cased bert_qa_bert_base_multilingual_cased_finetuned_chaii Question Answering English BertForQuestionAnswering
en.answer_question.chaii.bert.uncased bert_qa_bert_multi_uncased_finetuned_chaii Question Answering English BertForQuestionAnswering
en.answer_question.chaii.distil_bert distilbert_qa_multi_finetuned_for_xqua_on_chaii Question Answering English DistilBertForQuestionAnswering
en.answer_question.bio_clinical.bert bert_qa_sagemaker_BioclinicalBERT_ADR Question Answering English BertForQuestionAnswering
en.answer_question.roberta.by_LucasS roberta_qa_robertaBaseABSA Question Answering English RoBertaForQuestionAnswering
en.answer_question.korquad.bert.multilingual_base_cased.by_eliza-dukim bert_qa_bert_base_multilingual_cased_korquad_v1 Question Answering English BertForQuestionAnswering
en.answer_question.roberta.by_Nakul24 roberta_qa_RoBERTa_emotion_extraction Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.bert.large.by_rsvp-ai bert_qa_bertserini_bert_large_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.large.by_ruselkomp bert_qa_sbert_large_nlu_ru_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.large_cased bert_qa_bert_large_cased_whole_word_masking_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.large_uncased.by_Graphcore bert_qa_Graphcore_bert_large_uncased_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.large_uncased.by_haddadalwi bert_qa_bert_large_uncased_whole_word_masking_finetuned_squad_finetuned_islamic_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.large_uncased.by_howey bert_qa_howey_bert_large_uncased_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.large_uncased.by_internetoftim bert_qa_internetoftim_bert_large_uncased_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.large_uncased.by_ofirzaf bert_qa_ofirzaf_bert_large_uncased_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.large_uncased.by_uploaded by huggingface bert_qa_bert_large_uncased_whole_word_masking_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.large_uncased_sparse_80_1x4_block_pruneofa.by_Intel bert_qa_bert_large_uncased_squadv1.1_sparse_80_1x4_block_pruneofa Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.large_uncased_sparse_90_unstructured.by_Intel bert_qa_bert_large_uncased_squadv1.1_sparse_90_unstructured Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.medium.by_anas-awadalla bert_qa_bert_medium_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.medium.by_mrm8488 bert_qa_bert_medium_wrslb_finetuned_squadv1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.medium_finetuned.by_anas-awadalla bert_qa_bert_medium_pretrained_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.mini_lm_base_uncased bert_qa_MiniLM_L12_H384_uncased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.ms_tuned.base.by_zhufy bert_qa_squad_ms_bert_base Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.multilingual_base_cased.by_Paul-Vinh bert_qa_Paul_Vinh_bert_base_multilingual_cased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.multilingual_base_cased.by_salti bert_qa_salti_bert_base_multilingual_cased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.multilingual_base_cased.by_vanichandna bert_qa_bert_base_multilingual_cased_finetuned_squadv1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.multilingual_base_uncased bert_qa_bert_base_multilingual_uncased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.sl256.by_vuiseng9 bert_qa_bert_l_squadv1.1_sl256 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.distilled_base_uncased.by_kamilali bert_qa_kamilali_distilbert_base_uncased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.distilled_base_uncased.by_juliusco bert_qa_juliusco_distilbert_base_uncased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.distilled_base_uncased.by_huggingface bert_qa_prunebert_base_uncased_6_finepruned_w_distil_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.cased bert_qa_bert_multi_cased_squad_sv_marbogusz Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_Ghost1 bert_qa_bert_finetuned_squad1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_Harsit bert_qa_Harsit_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_KevinChoi bert_qa_KevinChoi_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_Kutay bert_qa_fine_tuned_squad_aip Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_Laikokwei bert_qa_Laikokwei_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_Neulvo bert_qa_Neulvo_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_andresestevez bert_qa_andresestevez_bert_finetuned_squad_accelerate Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_ankitkupadhyay bert_qa_ankitkupadhyay_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_datauma bert_qa_datauma_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_hendrixcosta bert_qa_bertimbau_squad1.1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.sl384.by_vuiseng9 bert_qa_bert_l_squadv1.1_sl384 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_huggingface-course bert_qa_huggingface_course_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_maroo93 bert_qa_squad1.1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_mrbalazs5 bert_qa_mrbalazs5_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_mrp bert_qa_mrp_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_nickmuchi bert_qa_nickmuchi_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_peterhsu bert_qa_tf_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_ruselkomp bert_qa_tests_finetuned_squad_test_bert Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_spacemanidol bert_qa_neuralmagic_bert_squad_12layer_0sparse Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_stevemobs bert_qa_bert_finetuned_squad_pytorch Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_vanichandna bert_qa_muril_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_youngjae bert_qa_youngjae_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_jatinshah bert_qa_jatinshah_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_FardinSaboori bert_qa_FardinSaboori_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.small.by_anas-awadalla bert_qa_bert_small_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.small_finetuned.by_anas-awadalla bert_qa_bert_small_pretrained_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_ParulChaudhari distilbert_qa_ParulChaudhari_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Plimpton distilbert_qa_Plimpton_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Raphaelg9 distilbert_qa_Raphaelg9_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Rocketknight1 distilbert_qa_Rocketknight1_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_SEISHIN distilbert_qa_SEISHIN_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Shashidhar distilbert_qa_Shashidhar_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Sourabh714 distilbert_qa_Sourabh714_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_SupriyaArun distilbert_qa_SupriyaArun_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Thitaree distilbert_qa_Thitaree_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Tianle distilbert_qa_Tianle_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_V3RX2000 distilbert_qa_V3RX2000_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Wiam distilbert_qa_Wiam_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_aaraki distilbert_qa_aaraki_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_abhinavkulkarni distilbert_qa_abhinavkulkarni_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_akr distilbert_qa_akr_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_andi611 distilbert_qa_andi611_base_uncased_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_anurag0077 distilbert_qa_base_uncased_finetuned_squad3 Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_arvalinno distilbert_qa_arvalinno_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_avioo1 distilbert_qa_avioo1_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_bdickson distilbert_qa_bdickson_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_caiosantillo distilbert_qa_caiosantillo_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Nadhiya distilbert_qa_Nadhiya_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_MYX4567 distilbert_qa_MYX4567_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_HomayounSadri distilbert_qa_HomayounSadri_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Hoang distilbert_qa_Hoang_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.bert.tiny bert_qa_bert_tiny_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.v1.1.by_maroo93 bert_qa_squad1.1_1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.v1.by_vanichandna bert_qa_muril_finetuned_squadv1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.v2.by_peterhsu bert_qa_peterhsu_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.v2.by_ruselkomp bert_qa_tests_finetuned_squad_test_bert_2 Question Answering English BertForQuestionAnswering
en.answer_question.squad.biobert.base_cased.by_dmis-lab bert_qa_biobert_base_cased_v1.1_squad Question Answering English BertForQuestionAnswering
en.answer_question.roberta.by_Mr-Wick roberta_qa_Roberta Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.bioformer.cased bert_qa_bioformer_cased_v1.0_squad1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.covid_bert bert_qa_covidbert_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.covid_biobert.base_cased bert_qa_biobert_base_cased_v1.1_squad_finetuned_covbiobert Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.small.by_mrm8488 bert_qa_bert_small_wrslb_finetuned_squadv1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.covid_roberta.base_cased bert_qa_biobert_base_cased_v1.1_squad_finetuned_covdrobert Question Answering English BertForQuestionAnswering
en.answer_question.squad.distil_bert.base_cased.by_ncduy distilbert_qa_base_cased_distilled_squad_finetuned_squad_test Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_cased.by_uploaded by huggingface distilbert_qa_base_cased_distilled_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_small_cased distilbert_qa_base_cased_distilled_squad_finetuned_squad_small Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_tiny_cased distilbert_qa_tiny_base_cased_distilled_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_21iridescent distilbert_qa_21iridescent_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Adrian distilbert_qa_Adrian_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Ayoola distilbert_qa_Ayoola_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_FOFer distilbert_qa_FOFer_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Firat distilbert_qa_Firat_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base_uncased.by_Gayathri distilbert_qa_Gayathri_base_uncased_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.distil_bert.base distilbert_qa_base_finetuned_squad Question Answering English DistilBertForQuestionAnswering
en.answer_question.squad.bert.by_Danastos bert_qa_squad_bert_el_Danastos Question Answering English BertForQuestionAnswering
en.answer_question.squad.biobert.base_cased.by_juliusco bert_qa_biobert_base_cased_v1.1_squad_finetuned_biobert Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_ArpanZS bert_qa_debug_squad Question Answering English BertForQuestionAnswering
en.answer_question.roberta.techqa_cline_emanuals.by_AnonymousSub roberta_qa_cline_emanuals_techqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.techqa_declutr.by_AnonymousSub roberta_qa_declutr_techqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.techqa_declutr_emanuals.by_AnonymousSub roberta_qa_declutr_emanuals_techqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.testabsa.by_eAsyle roberta_qa_testABSA Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.testabsa3.by_eAsyle roberta_qa_testABSA3 Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.tiny_768d roberta_qa_tinyroberta_6l_768d Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.unaugv3.by_comacrae roberta_qa_roberta_unaugv3 Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta_absa roberta_qa_robertaABSA Question Answering English RoBertaForQuestionAnswering
en.answer_question.scibert.v2 bert_qa_nolog_SciBert_v2 Question Answering English BertForQuestionAnswering
en.answer_question.span_bert.by_Nakul24 bert_qa_Spanbert_emotion_extraction Question Answering English BertForQuestionAnswering
en.answer_question.span_bert.by_manishiitg bert_qa_spanbert_recruit_qa Question Answering English BertForQuestionAnswering
en.answer_question.span_bert.large bert_qa_spanbert_large_recruit_qa Question Answering English BertForQuestionAnswering
en.answer_question.sqac.bert.base_cased bert_qa_bert_base_spanish_wwm_cased_finetuned_qa_sqac Question Answering English BertForQuestionAnswering
en.answer_question.sqac.bert.base_uncased bert_qa_bert_base_spanish_wwm_uncased_finetuned_qa_sqac Question Answering English BertForQuestionAnswering
en.answer_question.squad.albert.base_v2 albert_qa_base_v2_squad Question Answering English AlbertForQuestionAnswering
en.answer_question.squad.albert.by_SS8 albert_qa_squad_2.0 Question Answering English AlbertForQuestionAnswering
en.answer_question.squad.albert.xl albert_qa_xlarge_finetuned_squad Question Answering English AlbertForQuestionAnswering
en.answer_question.squad.albert.xxl albert_qa_xxlarge_finetuned_squad Question Answering English AlbertForQuestionAnswering
en.answer_question.squad.bert.accelerate.by_KevinChoi bert_qa_KevinChoi_bert_finetuned_squad_accelerate Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.accelerate.by_huggingface-course bert_qa_huggingface_course_bert_finetuned_squad_accelerate Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.accelerate.by_peterhsu bert_qa_peterhsu_bert_finetuned_squad_accelerate Question Answering English BertForQuestionAnswering
en.answer_question.roberta.techqa_cline.by_AnonymousSub roberta_qa_cline_techqa Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.paraphrasev3.by_comacrae roberta_qa_roberta_paraphrasev3 Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.large_seed_4 roberta_qa_roberta_large_data_seed_4 Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.large_seed_0.by_anas-awadalla roberta_qa_roberta_large_data_seed_0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.bert.by_DaisyMak bert_qa_bert_finetuned_squad_accelerate_10epoch_transformerfrozen Question Answering English BertForQuestionAnswering
en.answer_question.roberta.by_aravind-812 roberta_qa_roberta_train_json Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_arjunth2001 roberta_qa_priv_qna Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_billfrench roberta_qa_cyberlandr_door Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_nlpunibo roberta_qa_nlpunibo_roberta Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_pierrerappolt roberta_qa_cart Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_shmuelamar roberta_qa_REQA_RoBERTa Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_stevemobs roberta_qa_quales_iberlef Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_sunitha roberta_qa_roberta_customds_finetune Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_veronica320 roberta_qa_QA_for_Event_Extraction Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.bert.accelerate.by_youngjae bert_qa_youngjae_bert_finetuned_squad_accelerate Question Answering English BertForQuestionAnswering
en.answer_question.roberta.by_vesteinn roberta_qa_IceBERT_QA Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.ch_tuned.by_Gantenbein roberta_qa_ADDI_CH_RoBERTa Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.cv_custom_ds.by_sunitha roberta_qa_CV_Custom_DS Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.cv_merge_ds.by_sunitha roberta_qa_CV_Merge_DS Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.de_tuned.by_Gantenbein roberta_qa_ADDI_DE_RoBERTa Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.eda_and_parav3.by_comacrae roberta_qa_roberta_eda_and_parav3 Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.edav3.by_comacrae roberta_qa_roberta_edav3 Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.fi_tuned.by_Gantenbein roberta_qa_ADDI_FI_RoBERTa Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.fr_tuned.by_Gantenbein roberta_qa_ADDI_FR_RoBERTa Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.it_tuned.by_Gantenbein roberta_qa_ADDI_IT_RoBERTa Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.large_init_large_seed_0.by_anas-awadalla roberta_qa_roberta_large_initialization_seed_0 Question Answering English RoBertaForQuestionAnswering
en.answer_question.roberta.by_z-uo roberta_qa_roberta_qasper Question Answering English RoBertaForQuestionAnswering
en.answer_question.squad.bert.augmented bert_qa_augmented_Squad_Translated Question Answering English BertForQuestionAnswering
en.answer_question.squad.albert.by_rowan1224 albert_qa_squad_slp Question Answering English AlbertForQuestionAnswering
en.answer_question.squad.bert.base.by_rsvp-ai bert_qa_bertserini_bert_base_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_vuiseng9 bert_qa_vuiseng9_bert_base_uncased_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.x1.16_f88.1_d8_unstruct.by_madlag bert_qa_bert_base_uncased_squadv1_x1.16_f88.1_d8_unstruct_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_1024d_seed_42 bert_qa_bert_base_uncased_few_shot_k_1024_finetuned_squad_seed_42 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_128d_seed_0 bert_qa_bert_base_uncased_few_shot_k_128_finetuned_squad_seed_0 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base.by_mrm8488 bert_qa_bert_mini_wrslb_finetuned_squadv1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_1_block_sparse_0.20_v1.by_madlag bert_qa_bert_base_uncased_squad1.1_block_sparse_0.13_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_256d_seed_0 bert_qa_bert_base_uncased_few_shot_k_256_finetuned_squad_seed_0 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_32d_seed_0 bert_qa_bert_base_uncased_few_shot_k_32_finetuned_squad_seed_0 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_512d_seed_0 bert_qa_bert_base_uncased_few_shot_k_512_finetuned_squad_seed_0 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_64d_seed_0 bert_qa_bert_base_uncased_few_shot_k_64_finetuned_squad_seed_0 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_victoraavila bert_qa_victoraavila_bert_base_uncased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_l3.by_howey bert_qa_bert_base_uncased_squad_L3 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_seed_42 bert_qa_bert_base_uncased_few_shot_k_16_finetuned_squad_seed_42 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_v2.by_ericRosello bert_qa_bert_base_uncased_finetuned_squad_frozen_v2 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_v2.by_madlag bert_qa_bert_base_uncased_squad1.1_pruned_x3.2_v2 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_x1.16_f88.1_d8_unstruct_v1.by_madlag bert_qa_bert_base_uncased_squad1.1_block_sparse_0.20_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_x1.84_f88.7_d36_hybrid_filled_v1.by_madlag bert_qa_bert_base_uncased_squadv1_x1.96_f88.3_d27_hybrid_filled_opt_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_x1.96_f88.3_d27_hybrid_filled_opt_v1.by_madlag bert_qa_bert_base_uncased_squadv1_x2.01_f89.2_d30_hybrid_rewind_opt_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_x2.01_f89.2_d30_hybrid_rewind_opt_v1.by_madlag bert_qa_bert_base_uncased_squadv1_x2.32_f86.6_d15_hybrid_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_x2.32_f86.6_d15_hybrid_v1.by_madlag bert_qa_bert_base_uncased_squadv1_x2.44_f87.7_d26_hybrid_filled_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_x2.44_f87.7_d26_hybrid_filled_v1.by_madlag bert_qa_bert_base_uncased_squad1.1_block_sparse_0.07_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.by_Alexander-Learn bert_qa_Alexander_Learn_bert_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_l6.by_howey bert_qa_bert_base_uncased_squad_L6 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_tli8hf bert_qa_unqover_bert_base_uncased_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased_1_block_sparse_0.13_v1.by_madlag bert_qa_bert_base_uncased_squadv1_x1.84_f88.7_d36_hybrid_filled_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_madlag bert_qa_bert_base_uncased_squad_v1_sparse0.25 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base.by_vuiseng9 bert_qa_bert_base_squadv1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base.by_xraychen bert_qa_squad_baseline Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base.by_zhufy bert_qa_squad_en_bert_base Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_srmukundb bert_qa_srmukundb_bert_base_uncased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_cased.by_Seongkyu bert_qa_Seongkyu_bert_base_cased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_cased.by_SreyanG-NVIDIA bert_qa_SreyanG_NVIDIA_bert_base_cased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_cased.by_andresestevez bert_qa_andresestevez_bert_base_cased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_cased.by_batterydata bert_qa_bert_base_cased_squad_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_cased.by_ncduy bert_qa_bert_base_cased_finetuned_squad_test Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.1.1_block_sparse_0.32_v1.by_madlag bert_qa_bert_base_uncased_squad1.1_block_sparse_0.32_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_cased.by_KB bert_qa_bert_base_swedish_cased_squad_experimental Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_Intel bert_qa_bert_base_uncased_squadv1.1_sparse_80_1x4_block_pruneofa Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_lewtun bert_qa_bert_base_uncased_finetuned_squad_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_HomayounSadri bert_qa_HomayounSadri_bert_base_uncased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_kaporter bert_qa_kaporter_bert_base_uncased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_jgammack bert_qa_MTL_bert_base_uncased_ww_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_csarron bert_qa_csarron_bert_base_uncased_squad_v1 Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_jimypbr bert_qa_jimypbr_bert_base_uncased_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_bdickson bert_qa_bdickson_bert_base_uncased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_Tianle bert_qa_Tianle_bert_base_uncased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_SupriyaArun bert_qa_SupriyaArun_bert_base_uncased_finetuned_squad Question Answering English BertForQuestionAnswering
en.answer_question.squad.bert.base_uncased.by_SreyanG-NVIDIA bert_qa_SreyanG_NVIDIA_bert_base_uncased_finetuned_squad Question Answering English BertForQuestionAnswering
fi.answer_question.xlm_roberta xlm_roberta_qa_ADDI_FI_XLM_R Question Answering Finnish XlmRoBertaForQuestionAnswering
fr.answer_question.squad.xlmr_roberta.base xlm_roberta_qa_xlmr_base_texas_squad_fr_fr_saattrupdan Question Answering French XlmRoBertaForQuestionAnswering
de.answer_question.squad_spanish_tuned.xlmr_roberta.base.by_saattrupdan xlm_roberta_qa_xlmr_base_texas_squad_es_es_saattrupdan Question Answering German XlmRoBertaForQuestionAnswering
de.answer_question.xlm_roberta.base xlm_roberta_qa_xlm_roberta_base_german Question Answering German XlmRoBertaForQuestionAnswering
de.answer_question.squadv2.electra.base electra_qa_base_squad2 Question Answering German BertForQuestionAnswering
de.answer_question.squadv2.bert bert_qa_bert_multi_english_german_squad2 Question Answering German BertForQuestionAnswering
de.answer_question.squad_de_tuned.xlmr_roberta.base.by_saattrupdan xlm_roberta_qa_xlmr_base_texas_squad_de_de_saattrupdan Question Answering German XlmRoBertaForQuestionAnswering
de.answer_question.xlm_roberta xlm_roberta_qa_ADDI_DE_XLM_R Question Answering German XlmRoBertaForQuestionAnswering
de.answer_question.electra.distilled_base electra_qa_g_base_germanquad_distilled Question Answering German BertForQuestionAnswering
de.answer_question.electra.base electra_qa_g_base_germanquad Question Answering German BertForQuestionAnswering
de.answer_question.electra electra_qa_German_question_answer Question Answering German BertForQuestionAnswering
de.answer_question.bert bert_qa_GBERTQnA Question Answering German BertForQuestionAnswering
de.answer_question.electra.large electra_qa_g_large_germanquad Question Answering German BertForQuestionAnswering
he.answer_question.squad.bert bert_qa_hebert_finetuned_hebrew_squad Question Answering Hebrew BertForQuestionAnswering
hi.answer_question.xlm_roberta xlm_roberta_qa_autonlp_hindi_question_answering_23865268 Question Answering Hindi XlmRoBertaForQuestionAnswering
hi.answer_question.xlm_roberta.base xlm_roberta_qa_xlm_roberta_base_hindi Question Answering Hindi XlmRoBertaForQuestionAnswering
hu.answer_question.squad.bert bert_qa_huBert_fine_tuned_hungarian_squadv1 Question Answering Hungarian BertForQuestionAnswering
is.answer_question.squad.roberta roberta_qa_icebert_texas_squad_is_saattrupdan Question Answering Icelandic RoBertaForQuestionAnswering
is.answer_question.squad.xlmr_roberta.base xlm_roberta_qa_xlmr_base_texas_squad_is_is_saattrupdan Question Answering Icelandic XlmRoBertaForQuestionAnswering
is.answer_question.xlmr_roberta xlm_roberta_qa_XLMr_ENIS_QA_Is Question Answering Icelandic XlmRoBertaForQuestionAnswering
id.answer_question.indo_bert bert_qa_Indobert_QA Question Answering Indonesian BertForQuestionAnswering
it.answer_question.squad.bert bert_qa_bert_italian_finedtuned_squadv1_it_alfa Question Answering Italian BertForQuestionAnswering
it.answer_question.squad.bert.base_uncased bert_qa_bert_base_italian_uncased_squad_it_antoniocappiello Question Answering Italian BertForQuestionAnswering
it.answer_question.squad.bert.xxl_cased bert_qa_squad_xxl_cased_hub1 Question Answering Italian BertForQuestionAnswering
it.answer_question.xlm_roberta xlm_roberta_qa_ADDI_IT_XLM_R Question Answering Italian XlmRoBertaForQuestionAnswering
ja.answer_question.wikipedia.bert.base bert_qa_base_japanese_wikipedia_ud_head Question Answering Japanese BertForQuestionAnswering
ja.answer_question.wikipedia.bert.large bert_qa_large_japanese_wikipedia_ud_head Question Answering Japanese BertForQuestionAnswering
ko.answer_question.korquad.electra.small electra_qa_small_v3_finetuned_korquad Question Answering Korean BertForQuestionAnswering
ko.answer_question.korquad.electra.base_v2_384.by_monologg electra_qa_base_v2_finetuned_korquad_384 Question Answering Korean BertForQuestionAnswering
ko.answer_question.korquad.electra.base_v2.by_monologg electra_qa_base_v2_finetuned_korquad Question Answering Korean BertForQuestionAnswering
ko.answer_question.korquad.electra.base electra_qa_base_v3_finetuned_korquad Question Answering Korean BertForQuestionAnswering
ko.answer_question.klue.electra.base.by_seongju electra_qa_klue_mrc_base Question Answering Korean BertForQuestionAnswering
ko.answer_question.klue.bert.base.by_bespin-global bert_qa_bespin_global_klue_bert_base_mrc Question Answering Korean BertForQuestionAnswering
ko.answer_question.klue.bert.base_aihub.by_bespin-global bert_qa_klue_bert_base_aihub_mrc Question Answering Korean BertForQuestionAnswering
ko.answer_question.klue.bert.base.by_ainize bert_qa_ainize_klue_bert_base_mrc Question Answering Korean BertForQuestionAnswering
ko.answer_question.electra electra_qa_long Question Answering Korean BertForQuestionAnswering
ko.answer_question.klue.electra.base.by_obokkkk electra_qa_base_v3_discriminator_finetuned_klue_v4 Question Answering Korean BertForQuestionAnswering
el.answer_question.bert bert_qa_qacombination_bert_el_Danastos Question Answering Modern Greek (1453-) BertForQuestionAnswering
pl.answer_question.squad.bert.multilingual_base_cased bert_qa_bert_base_multilingual_cased_finetuned_polish_squad1 Question Answering Polish BertForQuestionAnswering
pl.answer_question.squadv2.bert.multilingual_base_cased bert_qa_bert_base_multilingual_cased_finetuned_polish_squad2 Question Answering Polish BertForQuestionAnswering
pt.answer_question.squad.distil_bert distilbert_qa_multi_finedtuned_squad Question Answering Portuguese DistilBertForQuestionAnswering
pt.answer_question.squad.bert.large_cased bert_qa_bert_large_cased_squad_v1.1_portuguese Question Answering Portuguese BertForQuestionAnswering
pt.answer_question.squad.biobert bert_qa_bioBERTpt_squad_v1.1_portuguese Question Answering Portuguese BertForQuestionAnswering
pt.answer_question.squad.bert.base_cased.by_mrm8488 bert_qa_bert_base_portuguese_cased_finetuned_squad_v1_pt_mrm8488 Question Answering Portuguese BertForQuestionAnswering
pt.answer_question.squad.bert.base_cased.by_pierreguillou bert_qa_bert_base_cased_squad_v1.1_portuguese Question Answering Portuguese BertForQuestionAnswering
ru.answer_question.distil_bert distilbert_qa_model_QA_5_epoch_RU Question Answering Russian DistilBertForQuestionAnswering
si.answer_question.bert.base bert_qa_bert_base_sinhala_qa Question Answering Sinhala, Sinhalese BertForQuestionAnswering
sv.answer_question.squadv2.bert.base bert_qa_bert_base_swedish_squad2 Question Answering Swedish BertForQuestionAnswering
sv.answer_question.xlmr_roberta.large xlm_roberta_qa_xlmr_large_qa_sv_sv_m3hrdadfi Question Answering Swedish XlmRoBertaForQuestionAnswering
ta.answer_question.squad.xlm_roberta xlm_roberta_qa_xlm_roberta_squad_tamil Question Answering Tamil XlmRoBertaForQuestionAnswering
th.answer_question.xquad_squad.bert.cased bert_qa_thai_bert_multi_cased_finetuned_xquadv1_finetuned_squad Question Answering Thai BertForQuestionAnswering
th.answer_question.xquad.multi_lingual_bert.base bert_qa_xquad_th_mbert_base Question Answering Thai BertForQuestionAnswering
th.answer_question.bert.multilingual_base_cased bert_qa_bert_base_multilingual_cased_finetune_qa Question Answering Thai BertForQuestionAnswering
th.answer_question.squadv2.xlm_roberta.base xlm_roberta_qa_thai_xlm_roberta_base_squad2 Question Answering Thai XlmRoBertaForQuestionAnswering
tr.answer_question.squad.electra electra_qa_enelpi_squad Question Answering Turkish BertForQuestionAnswering
tr.answer_question.xlm_roberta xlm_roberta_qa_XLM_Turkish Question Answering Turkish XlmRoBertaForQuestionAnswering
tr.answer_question.squadv2.electra.base_v2 electra_qa_base_discriminator_finetuned_squadv2 Question Answering Turkish BertForQuestionAnswering
tr.answer_question.squad.electra.base electra_qa_base_discriminator_finetuned_squadv1 Question Answering Turkish BertForQuestionAnswering
tr.answer_question.squad.bert.base bert_qa_bert_base_turkish_squad Question Answering Turkish BertForQuestionAnswering
tr.answer_question.bert.base_uncased bert_qa_loodos_bert_base_uncased_QA_fine_tuned Question Answering Turkish BertForQuestionAnswering
tr.answer_question.electra electra_qa_turkish Question Answering Turkish BertForQuestionAnswering
tr.answer_question.bert.distilled bert_qa_distilbert_tr_q_a Question Answering Turkish BertForQuestionAnswering
tr.answer_question.bert.by_yunusemreemik bert_qa_logo_qna_model Question Answering Turkish BertForQuestionAnswering
tr.answer_question.bert.by_lserinol bert_qa_bert_turkish_question_answering Question Answering Turkish BertForQuestionAnswering
tr.answer_question.electra.small_uncased electra_qa_small_turkish_uncased_discriminator_finetuned Question Answering Turkish BertForQuestionAnswering
uk.answer_question.xlmr_roberta xlmroberta_qa_ukrainian Question Answering Ukrainian XlmRoBertaForQuestionAnswering
vi.answer_question.xlm_roberta.large xlm_roberta_qa_xlm_roberta_large_vi_qa Question Answering Vietnamese XlmRoBertaForQuestionAnswering
ar.answer_question.bert bert_qa_arap_qa_bert Question Answering Arabic BertForQuestionAnswering
ar.answer_question.xlm_roberta.base xlm_roberta_qa_xlm_roberta_base_arabic Question Answering Arabic XlmRoBertaForQuestionAnswering
ar.answer_question.tydiqa.electra.base electra_qa_ara_base_artydiqa Question Answering Arabic BertForQuestionAnswering
ar.answer_question.squad_arcd.electra.base electra_qa_AraElectra_base_finetuned_ARCD Question Answering Arabic BertForQuestionAnswering
ar.answer_question.squad_arcd.electra.768d electra_qa_araElectra_SQUAD_ARCD_768 Question Answering Arabic BertForQuestionAnswering
ar.answer_question.xlm_roberta.large xlm_roberta_qa_xlm_roberta_large_arabic_qa Question Answering Arabic XlmRoBertaForQuestionAnswering
ar.answer_question.electra electra_qa_AraELECTRA_discriminator_SOQAL Question Answering Arabic BertForQuestionAnswering
ar.answer_question.bert.v2 bert_qa_arap_qa_bert_v2 Question Answering Arabic BertForQuestionAnswering
ar.answer_question.bert.large_v2 bert_qa_arap_qa_bert_large_v2 Question Answering Arabic BertForQuestionAnswering
ar.answer_question.squad_arcd.electra electra_qa_araElectra_SQUAD_ARCD Question Answering Arabic BertForQuestionAnswering
zh.answer_question.mac_bert.large bert_qa_chinese_pretrain_mrc_macbert_large Question Answering Chinese BertForQuestionAnswering
zh.answer_question.bert.multilingual_base_cased bert_qa_multilingual_bert_base_cased_chinese Question Answering Chinese BertForQuestionAnswering
zh.answer_question.bert.large.by_qalover bert_qa_chinese_pert_large_open_domain_mrc Question Answering Chinese BertForQuestionAnswering
zh.answer_question.bert.large.by_luhua bert_qa_chinese_pretrain_mrc_roberta_wwm_ext_large Question Answering Chinese BertForQuestionAnswering
zh.answer_question.bert.large.by_hfl bert_qa_chinese_pert_large_mrc Question Answering Chinese BertForQuestionAnswering
zh.answer_question.bert.base.by_uer bert_qa_roberta_base_chinese_extractive_qa Question Answering Chinese BertForQuestionAnswering
zh.answer_question.bert.by_jackh1995 bert_qa_bert_chinese_finetuned Question Answering Chinese BertForQuestionAnswering
zh.answer_question.bert.base.by_liam168 bert_qa_qa_roberta_base_chinese_extractive Question Answering Chinese BertForQuestionAnswering
zh.answer_question.bert.base.by_jackh1995 bert_qa_roberta_base_chinese_extractive_qa_scratch Question Answering Chinese BertForQuestionAnswering
zh.answer_question.bert.base.by_hfl bert_qa_chinese_pert_base_mrc Question Answering Chinese BertForQuestionAnswering
zh.answer_question.squad.bert.base bert_qa_bert_base_chinese_finetuned_squad_colab Question Answering Chinese BertForQuestionAnswering
zh.answer_question.bert.by_yechen bert_qa_question_answering_chinese Question Answering Chinese BertForQuestionAnswering
zh.answer_question.xlm_roberta.base xlm_roberta_qa_xlm_roberta_base_chinese Question Answering Chinese XlmRoBertaForQuestionAnswering
fa.answer_question.bert.base bert_qa_bert_base_fa_qa Question Answering Persian BertForQuestionAnswering
fa.answer_question.xlm_roberta.large xlm_roberta_qa_xlm_roberta_large_fa_qa Question Answering Persian XlmRoBertaForQuestionAnswering
fa.answer_question.xlmr_roberta.large xlmroberta_qa_xlmr_large Question Answering Persian XlmRoBertaForQuestionAnswering
sw.answer_question.tydiqa.xlm_roberta.base xlm_roberta_qa_afriberta_base_finetuned_tydiqa Question Answering Swahili (macrolanguage) XlmRoBertaForQuestionAnswering
vn.answer_question.xlm_roberta.base xlm_roberta_qa_xlm_roberta_base_vietnamese Question Answering nan XlmRoBertaForQuestionAnswering
xx.answer_question.chaii.xlm_roberta xlm_roberta_qa_xlm_roberta_qa_chaii Question Answering nan XlmRoBertaForQuestionAnswering
xx.answer_question.xquad.bert.uncased bert_qa_bert_multi_uncased_finetuned_xquadv1 Question Answering nan BertForQuestionAnswering
xx.answer_question.xquad.bert.cased bert_qa_bert_multi_cased_finetuned_xquadv1 Question Answering nan BertForQuestionAnswering
xx.answer_question.xlm_roberta.distilled xlm_roberta_qa_distill_xlm_mrc Question Answering nan XlmRoBertaForQuestionAnswering
xx.answer_question.tydiqa.multi_lingual_bert bert_qa_Part_1_mBERT_Model_E1 Question Answering nan BertForQuestionAnswering
xx.answer_question.tydiqa.bert bert_qa_telugu_bertu_tydiqa Question Answering nan BertForQuestionAnswering
xx.answer_question.xquad_tydiqa.bert.cased bert_qa_bert_multi_cased_finedtuned_xquad_tydiqa_goldp Question Answering nan BertForQuestionAnswering
xx.answer_question.squad.distil_bert._en_de_es_vi_zh_tuned.by_ZYW distilbert_qa_squad_en_de_es_vi_zh_model Question Answering nan DistilBertForQuestionAnswering
xx.answer_question.roberta roberta_qa_ft_lr_cu_leolin12345 Question Answering nan RoBertaForQuestionAnswering
xx.answer_question.distil_bert.vi_zh_es_tuned.by_ZYW distilbert_qa_en_de_vi_zh_es_model Question Answering nan DistilBertForQuestionAnswering
xx.answer_question.distil_bert.en_de_tuned.by_ZYW distilbert_qa_en_de_model Question Answering nan DistilBertForQuestionAnswering
xx.answer_question.distil_bert.en_de_es_tuned.by_ZYW distilbert_qa_en_de_es_model Question Answering nan DistilBertForQuestionAnswering
xx.answer_question.squad.distil_bert.en_de_es_tuned.by_ZYW distilbert_qa_squad_en_de_es_model Question Answering nan DistilBertForQuestionAnswering

Minor Improvements

  • IOB Schema Detection for Tokenclassifiers and adding NER Converting in those cases
  • Tweaks in column name generation of most annotators

Bug Fixes

  • fixed bug in multi lang parsing
  • fixed bug for Normalizers
  • fixed bug in fetching metadata for resolvers
  • fixed bug in deducting outputlevel and inferring output columns
  • fixed broken nlp_refs

NLU Version 3.4.4

600 new models with over 75 new languages including Ancient,Dead and Extinct languages, 155 languages total covered, 400% Tokenizer Speedup, 18x USE-Embeddings GPU speedup in John Snow Labs NLU 3.4.4

We are very excited to announce NLU 3.4.4 has been released with over 600 new model, over 75 new languages and 155 languages covered in total, 400% speedup for tokenizers and 18x speedup of UniversalSentenceEncoder on GPU. On the general NLP side we have transformer based Embeddings and Token Classifiers powered by state of the art CamemBertEmbeddings and DeBertaForTokenClassification based architectures as well as various new models for
Historical, Ancient,Dead, Extinct, Genetic and Constructed languages like Old Church Slavonic, Latin, Sanskrit, Esperanto, Volapük, Coptic, Nahuatl, Ancient Greek (to 1453), Old Russian. On the healthcare side we have Portuguese De-identification Models, have NER models for Gene detection and finally RxNorm Sentence resolution model for mapping and extracting pharmaceutical actions (e.g. analgesic, hypoglycemic) as well as treatments (e.g. backache, diabetes).

General NLP Models

All general NLP models

First time language models covered

The languages for these models are covered for the very first time ever by NLU.

Number Language Name(s) NLU Reference Spark NLP Reference Task Annotator Class ISO-639-1 ISO-639-2/639-5 ISO-639-3 Scope Language Type
0 Sanskrit sa.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel sa san san Individual Ancient
1 Sanskrit sa.lemma lemma_vedic Lemmatization LemmatizerModel sa san san Individual Ancient
2 Sanskrit sa.pos pos_vedic Part of Speech Tagging PerceptronModel sa san san Individual Ancient
3 Sanskrit sa.stopwords stopwords_iso Stop Words Removal StopWordsCleaner sa san san Individual Ancient
4 Volapük vo.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel vo vol vol Individual Constructed
5 Nahuatl languages nah.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nah nan Collective Genetic
6 Aragonese an.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel an arg arg Individual Living
7 Assamese as.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel as asm asm Individual Living
8 Asturian, Asturleonese, Bable, Leonese ast.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan ast ast Individual Living
9 Bashkir ba.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel ba bak bak Individual Living
10 Bavarian bar.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan bar Individual Living
11 Bishnupriya bpy.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan bpy Individual Living
12 Burmese my.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel my 639-2/T: mya639-2/B: bur mya Individual Living
13 Cebuano ceb.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan ceb ceb Individual Living
14 Central Bikol bcl.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan bcl Individual Living
15 Chechen ce.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel ce che che Individual Living
16 Chuvash cv.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel cv chv chv Individual Living
17 Corsican co.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel co cos cos Individual Living
18 Dhivehi, Divehi, Maldivian dv.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel dv div div Individual Living
19 Egyptian Arabic arz.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan arz Individual Living
20 Emiliano-Romagnolo eml.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel eml nan nan Individual Living
21 Erzya myv.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan myv myv Individual Living
22 Georgian ka.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel ka 639-2/T: kat639-2/B: geo kat Individual Living
23 Goan Konkani gom.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan gom Individual Living
24 Javanese jv.embed.distilbert distilbert_embeddings_javanese_distilbert_small Embeddings DistilBertEmbeddings jv jav jav Individual Living
25 Javanese jv.embed.javanese_distilbert_small_imdb distilbert_embeddings_javanese_distilbert_small_imdb Embeddings DistilBertEmbeddings jv jav jav Individual Living
26 Javanese jv.embed.javanese_roberta_small roberta_embeddings_javanese_roberta_small Embeddings RoBertaEmbeddings jv jav jav Individual Living
27 Javanese jv.embed.javanese_roberta_small_imdb roberta_embeddings_javanese_roberta_small_imdb Embeddings RoBertaEmbeddings jv jav jav Individual Living
28 Javanese jv.embed.javanese_bert_small_imdb bert_embeddings_javanese_bert_small_imdb Embeddings BertEmbeddings jv jav jav Individual Living
29 Javanese jv.embed.javanese_bert_small bert_embeddings_javanese_bert_small Embeddings BertEmbeddings jv jav jav Individual Living
30 Kirghiz, Kyrgyz ky.stopwords stopwords_iso Stop Words Removal StopWordsCleaner ky kir kir Individual Living
31 Letzeburgesch, Luxembourgish lb.stopwords stopwords_iso Stop Words Removal StopWordsCleaner lb ltz ltz Individual Living
32 Letzeburgesch, Luxembourgish lb.lemma lemma_spacylookup Lemmatization LemmatizerModel lb ltz ltz Individual Living
33 Letzeburgesch, Luxembourgish lb.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel lb ltz ltz Individual Living
34 Ligurian lij.stopwords stopwords_iso Stop Words Removal StopWordsCleaner nan nan lij Individual Living
35 Lombard lmo.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan lmo Individual Living
36 Low German, Low Saxon nds.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nds nds Individual Living
37 Macedonian mk.stopwords stopwords_iso Stop Words Removal StopWordsCleaner mk 639-2/T: mkd639-2/B: mac mkd Individual Living
38 Macedonian mk.lemma lemma_spacylookup Lemmatization LemmatizerModel mk 639-2/T: mkd639-2/B: mac mkd Individual Living
39 Macedonian mk.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel mk 639-2/T: mkd639-2/B: mac mkd Individual Living
40 Maithili mai.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan mai mai Individual Living
41 Manx gv.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel gv glv glv Individual Living
42 Mazanderani mzn.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan mzn Individual Living
43 Minangkabau min.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan min min Individual Living
44 Mingrelian xmf.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan xmf Individual Living
45 Mirandese mwl.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan mwl mwl Individual Living
46 Neapolitan nap.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nap nap Individual Living
47 Nepal Bhasa, Newari new.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan new new Individual Living
48 Northern Frisian frr.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan frr frr Individual Living
49 Northern Sami sme.lemma lemma_giella Lemmatization LemmatizerModel se sme sme Individual Living
50 Northern Sami sme.pos pos_giella Part of Speech Tagging PerceptronModel se sme sme Individual Living
51 Northern Sotho, Pedi, Sepedi nso.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nso nso Individual Living
52 Occitan (post 1500) oc.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel oc oci oci Individual Living
53 Ossetian, Ossetic os.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel os oss oss Individual Living
54 Pfaelzisch pfl.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan pfl Individual Living
55 Piemontese pms.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan pms Individual Living
56 Romansh rm.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel rm roh roh Individual Living
57 Scots sco.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan sco sco Individual Living
58 Sicilian scn.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan scn scn Individual Living
59 Sinhala, Sinhalese si.stopwords stopwords_iso Stop Words Removal StopWordsCleaner si sin sin Individual Living
60 Sinhala, Sinhalese si.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel si sin sin Individual Living
61 Sundanese su.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel su sun sun Individual Living
62 Sundanese su.embed.sundanese_roberta_base roberta_embeddings_sundanese_roberta_base Embeddings RoBertaEmbeddings su sun sun Individual Living
63 Tagalog tl.lemma lemma_spacylookup Lemmatization LemmatizerModel tl tgl tgl Individual Living
64 Tagalog tl.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel tl tgl tgl Individual Living
65 Tagalog tl.stopwords stopwords_iso Stop Words Removal StopWordsCleaner tl tgl tgl Individual Living
66 Tagalog tl.embed.roberta_tagalog_large roberta_embeddings_roberta_tagalog_large Embeddings RoBertaEmbeddings tl tgl tgl Individual Living
67 Tagalog tl.embed.roberta_tagalog_base roberta_embeddings_roberta_tagalog_base Embeddings RoBertaEmbeddings tl tgl tgl Individual Living
68 Tajik tg.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel tg tgk tgk Individual Living
69 Tatar tt.stopwords stopwords_iso Stop Words Removal StopWordsCleaner tt tat tat Individual Living
70 Tatar tt.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel tt tat tat Individual Living
71 Tigrinya ti.stopwords stopwords_iso Stop Words Removal StopWordsCleaner ti tir tir Individual Living
72 Tosk Albanian als.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan als Individual Living
73 Tswana tn.stopwords stopwords_iso Stop Words Removal StopWordsCleaner tn tsn tsn Individual Living
74 Turkmen tk.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel tk tuk tuk Individual Living
75 Upper Sorbian hsb.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan hsb hsb Individual Living
76 Venetian vec.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan vec Individual Living
77 Vlaams vls.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan vls Individual Living
78 Walloon wa.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel wa wln wln Individual Living
79 Waray (Philippines) war.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan war war Individual Living
80 Western Armenian hyw.pos pos_armtdp Part of Speech Tagging PerceptronModel nan nan hyw Individual Living
81 Western Armenian hyw.lemma lemma_armtdp Lemmatization LemmatizerModel nan nan hyw Individual Living
82 Western Frisian fy.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel fy fry fry Individual Living
83 Western Panjabi pnb.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan pnb Individual Living
84 Yakut sah.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan sah sah Individual Living
85 Zeeuws zea.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel nan nan zea Individual Living
86 Albanian sq.stopwords stopwords_iso Stop Words Removal StopWordsCleaner sq 639-2/T: sqi639-2/B: alb sqi Macrolanguage Living
87 Albanian sq.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel sq 639-2/T: sqi639-2/B: alb sqi Macrolanguage Living
88 Azerbaijani az.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel az aze aze Macrolanguage Living
89 Azerbaijani az.stopwords stopwords_iso Stop Words Removal StopWordsCleaner az aze aze Macrolanguage Living
90 Malagasy mg.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel mg mlg mlg Macrolanguage Living
91 Malay (macrolanguage) ms.embed.albert albert_embeddings_albert_large_bahasa_cased Embeddings AlbertEmbeddings ms 639-2/T: msa639-2/B: may msa Macrolanguage Living
92 Malay (macrolanguage) ms.embed.distilbert distilbert_embeddings_malaysian_distilbert_small Embeddings DistilBertEmbeddings ms 639-2/T: msa639-2/B: may msa Macrolanguage Living
93 Malay (macrolanguage) ms.embed.albert_tiny_bahasa_cased albert_embeddings_albert_tiny_bahasa_cased Embeddings AlbertEmbeddings ms 639-2/T: msa639-2/B: may msa Macrolanguage Living
94 Malay (macrolanguage) ms.embed.albert_base_bahasa_cased albert_embeddings_albert_base_bahasa_cased Embeddings AlbertEmbeddings ms 639-2/T: msa639-2/B: may msa Macrolanguage Living
95 Malay (macrolanguage) ms.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel ms 639-2/T: msa639-2/B: may msa Macrolanguage Living
96 Mongolian mn.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel mn mon mon Macrolanguage Living
97 Oriya (macrolanguage) or.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel or ori ori Macrolanguage Living
98 Pashto, Pushto ps.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel ps pus pus Macrolanguage Living
99 Quechua qu.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel qu que que Macrolanguage Living
100 Sardinian sc.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel sc srd srd Macrolanguage Living
101 Serbo-Croatian sh.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel sh nan nan Macrolanguage Living
102 Uzbek uz.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel uz uzb uzb Macrolanguage Living
All general NLP models

Powered by the incredible Spark NLP 3.4.4 and previous releases.

Number NLU Reference Spark NLP Reference Task Language Name(s) Annotator Class ISO-639-1 ISO-639-2/639-5 ISO-639-3 Language Type Scope
0 cu.pos pos_proiel Part of Speech Tagging Church Slavic, Church Slavonic, Old Bulgarian, Old Church Slavonic, Old Slavonic PerceptronModel cu chu chu Ancient Individual
1 la.lemma lemma_proiel Lemmatization Latin LemmatizerModel la lat lat Ancient Individual
2 la.lemma lemma_proiel Lemmatization Latin LemmatizerModel la lat lat Ancient Individual
3 la.pos pos_perseus Part of Speech Tagging Latin PerceptronModel la lat lat Ancient Individual
4 la.pos pos_perseus Part of Speech Tagging Latin PerceptronModel la lat lat Ancient Individual
5 sa.embed.w2v_cc_300d w2v_cc_300d Embeddings Sanskrit WordEmbeddingsModel sa san san Ancient Individual
6 sa.lemma lemma_vedic Lemmatization Sanskrit LemmatizerModel sa san san Ancient Individual
7 sa.pos pos_vedic Part of Speech Tagging Sanskrit PerceptronModel sa san san Ancient Individual
8 sa.stopwords stopwords_iso Stop Words Removal Sanskrit StopWordsCleaner sa san san Ancient Individual
9 eo.embed.w2v_cc_300d w2v_cc_300d Embeddings Esperanto WordEmbeddingsModel eo epo epo Constructed Individual
10 vo.embed.w2v_cc_300d w2v_cc_300d Embeddings Volapük WordEmbeddingsModel vo vol vol Constructed Individual
11 cop.pos pos_scriptorium Part of Speech Tagging Coptic PerceptronModel nan cop cop Extinct Individual
12 nah.embed.w2v_cc_300d w2v_cc_300d Embeddings Nahuatl languages WordEmbeddingsModel nan nah nan Genetic Collective
13 grc.lemma lemma_proiel Lemmatization Ancient Greek (to 1453) LemmatizerModel nan grc grc Historical Individual
14 grc.stopwords stopwords_iso Stop Words Removal Ancient Greek (to 1453) StopWordsCleaner nan grc grc Historical Individual
15 grc.lemma lemma_proiel Lemmatization Ancient Greek (to 1453) LemmatizerModel nan grc grc Historical Individual
16 grc.pos pos_proiel Part of Speech Tagging Ancient Greek (to 1453) PerceptronModel nan grc grc Historical Individual
17 orv.lemma lemma_torot Lemmatization Old Russian LemmatizerModel nan nan orv Historical Individual
18 af.embed.w2v_cc_300d w2v_cc_300d Embeddings Afrikaans WordEmbeddingsModel af afr afr Living Individual
19 af.stopwords stopwords_iso Stop Words Removal Afrikaans StopWordsCleaner af afr afr Living Individual
20 am.embed.w2v_cc_300d w2v_cc_300d Embeddings Amharic WordEmbeddingsModel am amh amh Living Individual
21 am.embed.am_roberta roberta_embeddings_am_roberta Embeddings Amharic RoBertaEmbeddings am amh amh Living Individual
22 am.stopwords stopwords_iso Stop Words Removal Amharic StopWordsCleaner am amh amh Living Individual
23 an.embed.w2v_cc_300d w2v_cc_300d Embeddings Aragonese WordEmbeddingsModel an arg arg Living Individual
24 hy.stopwords stopwords_iso Stop Words Removal Armenian StopWordsCleaner hy 639-2/T: hye639-2/B: arm hye Living Individual
25 hy.lemma lemma_armtdp Lemmatization Armenian LemmatizerModel hy 639-2/T: hye639-2/B: arm hye Living Individual
26 hy.embed.w2v_cc_300d w2v_cc_300d Embeddings Armenian WordEmbeddingsModel hy 639-2/T: hye639-2/B: arm hye Living Individual
27 as.embed.w2v_cc_300d w2v_cc_300d Embeddings Assamese WordEmbeddingsModel as asm asm Living Individual
28 ast.embed.w2v_cc_300d w2v_cc_300d Embeddings Asturian, Asturleonese, Bable, Leonese WordEmbeddingsModel nan ast ast Living Individual
29 ba.embed.w2v_cc_300d w2v_cc_300d Embeddings Bashkir WordEmbeddingsModel ba bak bak Living Individual
30 eu.stopwords stopwords_iso Stop Words Removal Basque StopWordsCleaner eu 639-2/T: eus639-2/B: baq eus Living Individual
31 eu.embed.w2v_cc_300d w2v_cc_300d Embeddings Basque WordEmbeddingsModel eu 639-2/T: eus639-2/B: baq eus Living Individual
32 eu.lemma lemma_bdt Lemmatization Basque LemmatizerModel eu 639-2/T: eus639-2/B: baq eus Living Individual
33 bar.embed.w2v_cc_300d w2v_cc_300d Embeddings Bavarian WordEmbeddingsModel nan nan bar Living Individual
34 be.embed.w2v_cc_300d w2v_cc_300d Embeddings Belarusian WordEmbeddingsModel be bel bel Living Individual
35 be.lemma lemma_hse Lemmatization Belarusian LemmatizerModel be bel bel Living Individual
36 bn.embed.indic_transformers_bn_distilbert distilbert_embeddings_indic_transformers_bn_distilbert Embeddings Bengali DistilBertEmbeddings bn ben ben Living Individual
37 bn.embed.w2v_cc_300d w2v_cc_300d Embeddings Bengali WordEmbeddingsModel bn ben ben Living Individual
38 bn.embed.indic_transformers_bn_bert bert_embeddings_indic_transformers_bn_bert Embeddings Bengali BertEmbeddings bn ben ben Living Individual
39 bn.embed.muril_adapted_local bert_embeddings_muril_adapted_local Embeddings Bengali BertEmbeddings bn ben ben Living Individual
40 bn.embed.bangla_bert bert_embeddings_bangla_bert Embeddings Bengali BertEmbeddings bn ben ben Living Individual
41 bn.stopwords stopwords_iso Stop Words Removal Bengali StopWordsCleaner bn ben ben Living Individual
42 bh.embed.w2v_cc_300d w2v_cc_300d Embeddings Bihari language group, also known ash bih in ISO 639-2/5 WordEmbeddingsModel bh nan nan Living Individual
43 bpy.embed.w2v_cc_300d w2v_cc_300d Embeddings Bishnupriya WordEmbeddingsModel nan nan bpy Living Individual
44 bs.embed.w2v_cc_300d w2v_cc_300d Embeddings Bosnian WordEmbeddingsModel bs bos bos Living Individual
45 br.embed.w2v_cc_300d w2v_cc_300d Embeddings Breton WordEmbeddingsModel br bre bre Living Individual
46 bg.embed.w2v_cc_300d w2v_cc_300d Embeddings Bulgarian WordEmbeddingsModel bg bul bul Living Individual
47 bg.stopwords stopwords_iso Stop Words Removal Bulgarian StopWordsCleaner bg bul bul Living Individual
48 my.embed.w2v_cc_300d w2v_cc_300d Embeddings Burmese WordEmbeddingsModel my 639-2/T: mya639-2/B: bur mya Living Individual
49 es.embed.distilbert_base_es_multilingual_cased distilbert_embeddings_distilbert_base_es_multilingual_cased Embeddings Castilian, Spanish DistilBertEmbeddings es spa spa Living Individual
50 es.embed.distilbert_base_es_cased distilbert_embeddings_distilbert_base_es_cased Embeddings Castilian, Spanish DistilBertEmbeddings es spa spa Living Individual
51 es.embed.bertin_base_gaussian roberta_embeddings_bertin_base_gaussian Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
52 es.embed.bertin_roberta_base_spanish roberta_embeddings_bertin_roberta_base_spanish Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
53 es.embed.bertin_roberta_large_spanish roberta_embeddings_bertin_roberta_large_spanish Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
54 es.embed.bertin_base_stepwise roberta_embeddings_bertin_base_stepwise Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
55 es.embed.dpr_spanish_passage_encoder_allqa_base bert_embeddings_dpr_spanish_passage_encoder_allqa_base Embeddings Castilian, Spanish BertEmbeddings es spa spa Living Individual
56 es.embed.dpr_spanish_question_encoder_allqa_base bert_embeddings_dpr_spanish_question_encoder_allqa_base Embeddings Castilian, Spanish BertEmbeddings es spa spa Living Individual
57 es.embed.beto_gn_base_cased bert_embeddings_beto_gn_base_cased Embeddings Castilian, Spanish BertEmbeddings es spa spa Living Individual
58 es.embed.dpr_spanish_passage_encoder_squades_base bert_embeddings_dpr_spanish_passage_encoder_squades_base Embeddings Castilian, Spanish BertEmbeddings es spa spa Living Individual
59 es.embed.dpr_spanish_question_encoder_squades_base bert_embeddings_dpr_spanish_question_encoder_squades_base Embeddings Castilian, Spanish BertEmbeddings es spa spa Living Individual
60 es.embed.bert_base_es_cased bert_embeddings_bert_base_es_cased Embeddings Castilian, Spanish BertEmbeddings es spa spa Living Individual
61 es.embed.bert_base_5lang_cased bert_embeddings_bert_base_5lang_cased Embeddings Castilian, Spanish BertEmbeddings es spa spa Living Individual
62 es.embed.alberti_bert_base_multilingual_cased bert_embeddings_alberti_bert_base_multilingual_cased Embeddings Castilian, Spanish BertEmbeddings es spa spa Living Individual
63 es.embed.roberta_base_bne roberta_embeddings_roberta_base_bne Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
64 es.embed.jurisbert roberta_embeddings_jurisbert Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
65 es.embed.mlm_spanish_roberta_base roberta_embeddings_mlm_spanish_roberta_base Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
66 es.embed.roberta_large_bne roberta_embeddings_roberta_large_bne Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
67 es.pos pos_ancora Part of Speech Tagging Castilian, Spanish PerceptronModel es spa spa Living Individual
68 es.embed.bertin_base_random_exp_512seqlen roberta_embeddings_bertin_base_random_exp_512seqlen Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
69 es.embed.bertin_base_gaussian_exp_512seqlen roberta_embeddings_bertin_base_gaussian_exp_512seqlen Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
70 es.ner.roberta_base_bne_capitel_ner_plus roberta_ner_roberta_base_bne_capitel_ner_plus Named Entity Recognition Castilian, Spanish RoBertaForTokenClassification es spa spa Living Individual
71 es.ner.roberta_base_bne_capitel_ner roberta_ner_roberta_base_bne_capitel_ner Named Entity Recognition Castilian, Spanish RoBertaForTokenClassification es spa spa Living Individual
72 es.ner.RuPERTa_base_finetuned_ner roberta_ner_RuPERTa_base_finetuned_ner Named Entity Recognition Castilian, Spanish RoBertaForTokenClassification es spa spa Living Individual
73 es.pos.roberta_base_bne_capitel_pos roberta_pos_roberta_base_bne_capitel_pos Part of Speech Tagging Castilian, Spanish RoBertaForTokenClassification es spa spa Living Individual
74 es.ner.NER_LAW_MONEY4 roberta_ner_NER_LAW_MONEY4 Named Entity Recognition Castilian, Spanish RoBertaForTokenClassification es spa spa Living Individual
75 es.pos.roberta_large_bne_capitel_pos roberta_pos_roberta_large_bne_capitel_pos Part of Speech Tagging Castilian, Spanish RoBertaForTokenClassification es spa spa Living Individual
76 es.ner.bsc_bio_ehr_es_pharmaconer roberta_ner_bsc_bio_ehr_es_pharmaconer Named Entity Recognition Castilian, Spanish RoBertaForTokenClassification es spa spa Living Individual
77 es.embed.RoBERTalex roberta_embeddings_RoBERTalex Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
78 es.ner.roberta_large_bne_capitel_ner roberta_ner_roberta_large_bne_capitel_ner Named Entity Recognition Castilian, Spanish RoBertaForTokenClassification es spa spa Living Individual
79 es.embed.RuPERTa_base roberta_embeddings_RuPERTa_base Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
80 es.embed.bertin_base_random roberta_embeddings_bertin_base_random Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
81 es.lemma lemma_spacylookup Lemmatization Castilian, Spanish LemmatizerModel es spa spa Living Individual
82 es.stopwords stopwords_iso Stop Words Removal Castilian, Spanish StopWordsCleaner es spa spa Living Individual
83 es.pos.RuPERTa_base_finetuned_pos roberta_pos_RuPERTa_base_finetuned_pos Part of Speech Tagging Castilian, Spanish RoBertaForTokenClassification es spa spa Living Individual
84 es.embed.bertin_base_stepwise_exp_512seqlen roberta_embeddings_bertin_base_stepwise_exp_512seqlen Embeddings Castilian, Spanish RoBertaEmbeddings es spa spa Living Individual
85 es.ner.bsc_bio_ehr_es_cantemist roberta_ner_bsc_bio_ehr_es_cantemist Named Entity Recognition Castilian, Spanish RoBertaForTokenClassification es spa spa Living Individual
86 ca.lemma lemma_spacylookup Lemmatization Catalan, Valencian LemmatizerModel ca cat cat Living Individual
87 ca.embed.w2v_cc_300d w2v_cc_300d Embeddings Catalan, Valencian WordEmbeddingsModel ca cat cat Living Individual
88 ca.stopwords stopwords_iso Stop Words Removal Catalan, Valencian StopWordsCleaner ca cat cat Living Individual
89 ceb.embed.w2v_cc_300d w2v_cc_300d Embeddings Cebuano WordEmbeddingsModel nan ceb ceb Living Individual
90 bcl.embed.w2v_cc_300d w2v_cc_300d Embeddings Central Bikol WordEmbeddingsModel nan nan bcl Living Individual
91 ce.embed.w2v_cc_300d w2v_cc_300d Embeddings Chechen WordEmbeddingsModel ce che che Living Individual
92 cv.embed.w2v_cc_300d w2v_cc_300d Embeddings Chuvash WordEmbeddingsModel cv chv chv Living Individual
93 co.embed.w2v_cc_300d w2v_cc_300d Embeddings Corsican WordEmbeddingsModel co cos cos Living Individual
94 hr.embed.w2v_cc_300d w2v_cc_300d Embeddings Croatian WordEmbeddingsModel hr hrv hrv Living Individual
95 hr.stopwords stopwords_iso Stop Words Removal Croatian StopWordsCleaner hr hrv hrv Living Individual
96 hr.lemma lemma_spacylookup Lemmatization Croatian LemmatizerModel hr hrv hrv Living Individual
97 cs.stopwords stopwords_iso Stop Words Removal Czech StopWordsCleaner cs 639-2/T: ces639-2/B: cze ces Living Individual
98 cs.embed.w2v_cc_300d w2v_cc_300d Embeddings Czech WordEmbeddingsModel cs 639-2/T: ces639-2/B: cze ces Living Individual
99 cs.pos pos_fictree Part of Speech Tagging Czech PerceptronModel cs 639-2/T: ces639-2/B: cze ces Living Individual
100 cs.lemma lemma_cltt Lemmatization Czech LemmatizerModel cs 639-2/T: ces639-2/B: cze ces Living Individual
101 cs.lemma lemma_cltt Lemmatization Czech LemmatizerModel cs 639-2/T: ces639-2/B: cze ces Living Individual
102 cs.lemma lemma_cltt Lemmatization Czech LemmatizerModel cs 639-2/T: ces639-2/B: cze ces Living Individual
103 da.embed.w2v_cc_300d w2v_cc_300d Embeddings Danish WordEmbeddingsModel da dan dan Living Individual
104 da.lemma lemma_spacylookup Lemmatization Danish LemmatizerModel da dan dan Living Individual
105 da.stopwords stopwords_iso Stop Words Removal Danish StopWordsCleaner da dan dan Living Individual
106 dv.embed.w2v_cc_300d w2v_cc_300d Embeddings Dhivehi, Divehi, Maldivian WordEmbeddingsModel dv div div Living Individual
107 nl.embed.distilbert_base_cased distilbert_embeddings_distilbert_base_nl_cased Embeddings Dutch, Flemish DistilBertEmbeddings nl 639-2/T: nld639-2/B: dut nld Living Individual
108 nl.pos.fullstop_dutch_punctuation_prediction roberta_pos_fullstop_dutch_punctuation_prediction Part of Speech Tagging Dutch, Flemish RoBertaForTokenClassification nl 639-2/T: nld639-2/B: dut nld Living Individual
109 nl.stopwords stopwords_iso Stop Words Removal Dutch, Flemish StopWordsCleaner nl 639-2/T: nld639-2/B: dut nld Living Individual
110 nl.embed.robbert_v2_dutch_base roberta_embeddings_robbert_v2_dutch_base Embeddings Dutch, Flemish RoBertaEmbeddings nl 639-2/T: nld639-2/B: dut nld Living Individual
111 nl.embed.robbertje_1_gb_bort roberta_embeddings_robbertje_1_gb_bort Embeddings Dutch, Flemish RoBertaEmbeddings nl 639-2/T: nld639-2/B: dut nld Living Individual
112 nl.embed.robbertje_1_gb_shuffled roberta_embeddings_robbertje_1_gb_shuffled Embeddings Dutch, Flemish RoBertaEmbeddings nl 639-2/T: nld639-2/B: dut nld Living Individual
113 nl.embed.robbertje_1_gb_non_shuffled roberta_embeddings_robbertje_1_gb_non_shuffled Embeddings Dutch, Flemish RoBertaEmbeddings nl 639-2/T: nld639-2/B: dut nld Living Individual
114 nl.embed.robbertje_1_gb_merged roberta_embeddings_robbertje_1_gb_merged Embeddings Dutch, Flemish RoBertaEmbeddings nl 639-2/T: nld639-2/B: dut nld Living Individual
115 nl.embed.w2v_cc_300d w2v_cc_300d Embeddings Dutch, Flemish WordEmbeddingsModel nl 639-2/T: nld639-2/B: dut nld Living Individual
116 nl.lemma lemma_spacylookup Lemmatization Dutch, Flemish LemmatizerModel nl 639-2/T: nld639-2/B: dut nld Living Individual
117 arz.embed.w2v_cc_300d w2v_cc_300d Embeddings Egyptian Arabic WordEmbeddingsModel nan nan arz Living Individual
118 eml.embed.w2v_cc_300d w2v_cc_300d Embeddings Emiliano-Romagnolo WordEmbeddingsModel eml nan nan Living Individual
119 en.ner.debertav3_large.conll03 deberta_v3_large_token_classifier_conll03 Named Entity Recognition English DeBertaForTokenClassification en eng eng Living Individual
120 en.ner.debertav3_base.conll03 deberta_v3_base_token_classifier_conll03 Named Entity Recognition English DeBertaForTokenClassification en eng eng Living Individual
121 en.ner.debertav3_small.conll03 deberta_v3_small_token_classifier_conll03 Named Entity Recognition English DeBertaForTokenClassification en eng eng Living Individual
122 en.ner.debertav3_xsmall.conll03 deberta_v3_xsmall_token_classifier_conll03 Named Entity Recognition English DeBertaForTokenClassification en eng eng Living Individual
123 en.ner.debertav3_large.ontonotes deberta_v3_large_token_classifier_ontonotes Named Entity Recognition English DeBertaForTokenClassification en eng eng Living Individual
124 en.ner.debertav3_base.ontonotes deberta_v3_base_token_classifier_ontonotes Named Entity Recognition English DeBertaForTokenClassification en eng eng Living Individual
125 en.ner.debertav3_small.ontonotes deberta_v3_small_token_classifier_ontonotes Named Entity Recognition English DeBertaForTokenClassification en eng eng Living Individual
126 en.ner.debertav3_xsmall.ontonotes deberta_v3_xsmall_token_classifier_ontonotes Named Entity Recognition English DeBertaForTokenClassification en eng eng Living Individual
127 en.med_ner.biomedical_bc2gm ner_biomedical_bc2gm Named Entity Recognition English MedicalNerModel en eng eng Living Individual
128 en.med_ner.biomedical_bc2gm ner_biomedical_bc2gm Named Entity Recognition English MedicalNerModel en eng eng Living Individual
129 en.resolve.rxnorm_action_treatment sbiobertresolve_rxnorm_action_treatment Entity Resolution English SentenceEntityResolverModel en eng eng Living Individual
130 en.embed.albert_xlarge_v1 albert_embeddings_albert_xlarge_v1 Embeddings English AlbertEmbeddings en eng eng Living Individual
131 en.embed.albert_base_v1 albert_embeddings_albert_base_v1 Embeddings English AlbertEmbeddings en eng eng Living Individual
132 en.embed.albert_xxlarge_v1 albert_embeddings_albert_xxlarge_v1 Embeddings English AlbertEmbeddings en eng eng Living Individual
133 en.embed.distilbert_base_en_cased distilbert_embeddings_distilbert_base_en_cased Embeddings English DistilBertEmbeddings en eng eng Living Individual
134 en.embed.distilbert_base_uncased_sparse_90_unstructured_pruneofa distilbert_embeddings_distilbert_base_uncased_sparse_90_unstructured_pruneofa Embeddings English DistilBertEmbeddings en eng eng Living Individual
135 en.embed.distilbert_base_uncased_sparse_85_unstructured_pruneofa distilbert_embeddings_distilbert_base_uncased_sparse_85_unstructured_pruneofa Embeddings English DistilBertEmbeddings en eng eng Living Individual
136 en.classify.questionpair classifierdl_electra_questionpair Text Classification English ClassifierDLModel en eng eng Living Individual
137 en.classify.question_vs_statement bert_sequence_classifier_question_statement Text Classification English BertForSequenceClassification en eng eng Living Individual
138 en.classify.song_lyrics bert_sequence_classifier_song_lyrics Text Classification English BertForSequenceClassification en eng eng Living Individual
139 en.embed.muppet_roberta_base roberta_embeddings_muppet_roberta_base Embeddings English RoBertaEmbeddings en eng eng Living Individual
140 en.embed.muppet_roberta_large roberta_embeddings_muppet_roberta_large Embeddings English RoBertaEmbeddings en eng eng Living Individual
141 en.embed.fairlex_ecthr_minilm roberta_embeddings_fairlex_ecthr_minilm Embeddings English RoBertaEmbeddings en eng eng Living Individual
142 en.embed.distilroberta_base_finetuned_jira_qt_issue_titles_and_bodies roberta_embeddings_distilroberta_base_finetuned_jira_qt_issue_titles_and_bodies Embeddings English RoBertaEmbeddings en eng eng Living Individual
143 en.embed.legal_roberta_base roberta_embeddings_legal_roberta_base Embeddings English RoBertaEmbeddings en eng eng Living Individual
144 en.embed.distilroberta_base roberta_embeddings_distilroberta_base Embeddings English RoBertaEmbeddings en eng eng Living Individual
145 en.embed.pmc_med_bio_mlm_roberta_large roberta_embeddings_pmc_med_bio_mlm_roberta_large Embeddings English RoBertaEmbeddings en eng eng Living Individual
146 en.lemma lemma_lines Lemmatization English LemmatizerModel en eng eng Living Individual
147 en.lemma lemma_lines Lemmatization English LemmatizerModel en eng eng Living Individual
148 en.lemma lemma_lines Lemmatization English LemmatizerModel en eng eng Living Individual
149 en.embed.roberta_pubmed roberta_embeddings_roberta_pubmed Embeddings English RoBertaEmbeddings en eng eng Living Individual
150 en.embed.fairlex_scotus_minilm roberta_embeddings_fairlex_scotus_minilm Embeddings English RoBertaEmbeddings en eng eng Living Individual
151 en.embed.distilroberta_base_finetuned_jira_qt_issue_title roberta_embeddings_distilroberta_base_finetuned_jira_qt_issue_title Embeddings English RoBertaEmbeddings en eng eng Living Individual
152 en.embed.chEMBL26_smiles_v2 roberta_embeddings_chEMBL26_smiles_v2 Embeddings English RoBertaEmbeddings en eng eng Living Individual
153 en.embed.SecRoBERTa roberta_embeddings_SecRoBERTa Embeddings English RoBertaEmbeddings en eng eng Living Individual
154 en.embed.distilroberta_base_climate_d_s roberta_embeddings_distilroberta_base_climate_d_s Embeddings English RoBertaEmbeddings en eng eng Living Individual
155 en.embed.chEMBL_smiles_v1 roberta_embeddings_chEMBL_smiles_v1 Embeddings English RoBertaEmbeddings en eng eng Living Individual
156 en.embed.distilroberta_base_climate_f roberta_embeddings_distilroberta_base_climate_f Embeddings English RoBertaEmbeddings en eng eng Living Individual
157 en.embed.distilroberta_base_climate_d roberta_embeddings_distilroberta_base_climate_d Embeddings English RoBertaEmbeddings en eng eng Living Individual
158 en.embed.Bible_roberta_base roberta_embeddings_Bible_roberta_base Embeddings English RoBertaEmbeddings en eng eng Living Individual
159 en.embed.w2v_cc_300d w2v_cc_300d Embeddings English WordEmbeddingsModel en eng eng Living Individual
160 en.pos pos_atis Part of Speech Tagging English PerceptronModel en eng eng Living Individual
161 en.ner.ner_chemical_bionlp_bc5cdr_pubmed roberta_ner_ner_chemical_bionlp_bc5cdr_pubmed Named Entity Recognition English RoBertaForTokenClassification en eng eng Living Individual
162 en.pos.roberta_large_english_upos roberta_pos_roberta_large_english_upos Part of Speech Tagging English RoBertaForTokenClassification en eng eng Living Individual
163 en.ner.roberta_ticker roberta_ner_roberta_ticker Named Entity Recognition English RoBertaForTokenClassification en eng eng Living Individual
164 en.embed.bert_political_election2020_twitter_mlm bert_embeddings_bert_political_election2020_twitter_mlm Embeddings English BertEmbeddings en eng eng Living Individual
165 en.embed.bert_base_uncased_mnli_sparse_70_unstructured_no_classifier bert_embeddings_bert_base_uncased_mnli_sparse_70_unstructured_no_classifier Embeddings English BertEmbeddings en eng eng Living Individual
166 en.embed.crosloengual_bert bert_embeddings_crosloengual_bert Embeddings English BertEmbeddings en eng eng Living Individual
167 en.embed.chemical_bert_uncased bert_embeddings_chemical_bert_uncased Embeddings English BertEmbeddings en eng eng Living Individual
168 en.embed.deberta_base_uncased bert_embeddings_deberta_base_uncased Embeddings English BertEmbeddings en eng eng Living Individual
169 en.embed.bert_base_en_cased bert_embeddings_bert_base_en_cased Embeddings English BertEmbeddings en eng eng Living Individual
170 en.embed.bert_for_patents bert_embeddings_bert_for_patents Embeddings English BertEmbeddings en eng eng Living Individual
171 en.embed.SecBERT bert_embeddings_SecBERT Embeddings English BertEmbeddings en eng eng Living Individual
172 en.embed.bert_base_5lang_cased bert_embeddings_bert_base_5lang_cased Embeddings English BertEmbeddings en eng eng Living Individual
173 en.embed.DiLBERT bert_embeddings_DiLBERT Embeddings English BertEmbeddings en eng eng Living Individual
174 en.embed.FinancialBERT bert_embeddings_FinancialBERT Embeddings English BertEmbeddings en eng eng Living Individual
175 en.embed.false_positives_scancode_bert_base_uncased_L8_1 bert_embeddings_false_positives_scancode_bert_base_uncased_L8_1 Embeddings English BertEmbeddings en eng eng Living Individual
176 en.embed.legal_bert_small_uncased bert_embeddings_legal_bert_small_uncased Embeddings English BertEmbeddings en eng eng Living Individual
177 en.embed.legal_bert_base_uncased bert_embeddings_legal_bert_base_uncased Embeddings English BertEmbeddings en eng eng Living Individual
178 en.embed.COVID_SciBERT bert_embeddings_COVID_SciBERT Embeddings English BertEmbeddings en eng eng Living Individual
179 en.embed.e bert_biolink_base Embeddings English BertEmbeddings en eng eng Living Individual
180 en.embed.danbert_small_cased bert_embeddings_danbert_small_cased Embeddings English BertEmbeddings en eng eng Living Individual
181 en.embed.bert_base_uncased_dstc9 bert_embeddings_bert_base_uncased_dstc9 Embeddings English BertEmbeddings en eng eng Living Individual
182 en.embed.hateBERT bert_embeddings_hateBERT Embeddings English BertEmbeddings en eng eng Living Individual
183 en.embed.childes_bert bert_embeddings_childes_bert Embeddings English BertEmbeddings en eng eng Living Individual
184 en.embed.clinical_pubmed_bert_base_512 bert_embeddings_clinical_pubmed_bert_base_512 Embeddings English BertEmbeddings en eng eng Living Individual
185 en.embed.netbert bert_embeddings_netbert Embeddings English BertEmbeddings en eng eng Living Individual
186 en.embed.psych_search bert_embeddings_psych_search Embeddings English BertEmbeddings en eng eng Living Individual
187 en.embed.muril_adapted_local bert_embeddings_muril_adapted_local Embeddings English BertEmbeddings en eng eng Living Individual
188 en.embed.finbert_pretrain_yiyanghkust bert_embeddings_finbert_pretrain_yiyanghkust Embeddings English BertEmbeddings en eng eng Living Individual
189 en.embed.lic_class_scancode_bert_base_cased_L32_1 bert_embeddings_lic_class_scancode_bert_base_cased_L32_1 Embeddings English BertEmbeddings en eng eng Living Individual
190 en.embed.sec_bert_sh bert_embeddings_sec_bert_sh Embeddings English BertEmbeddings en eng eng Living Individual
191 en.embed.sec_bert_num bert_embeddings_sec_bert_num Embeddings English BertEmbeddings en eng eng Living Individual
192 en.embed.finest_bert bert_embeddings_finest_bert Embeddings English BertEmbeddings en eng eng Living Individual
193 en.embed.bert_large_cased_whole_word_masking bert_embeddings_bert_large_cased_whole_word_masking Embeddings English BertEmbeddings en eng eng Living Individual
194 en.embed.clinical_pubmed_bert_base_128 bert_embeddings_clinical_pubmed_bert_base_128 Embeddings English BertEmbeddings en eng eng Living Individual
195 en.embed.bert_base_uncased_sparse_70_unstructured bert_embeddings_bert_base_uncased_sparse_70_unstructured Embeddings English BertEmbeddings en eng eng Living Individual
196 en.embed.sec_bert_base bert_embeddings_sec_bert_base Embeddings English BertEmbeddings en eng eng Living Individual
197 en.stopwords stopwords_iso Stop Words Removal English StopWordsCleaner en eng eng Living Individual
198 en.embed.agriculture_bert_uncased bert_embeddings_agriculture_bert_uncased Embeddings English BertEmbeddings en eng eng Living Individual
199 en.embed.bert_large_uncased_whole_word_masking bert_embeddings_bert_large_uncased_whole_word_masking Embeddings English BertEmbeddings en eng eng Living Individual
200 en.embed.ge bert_biolink_large Embeddings English BertEmbeddings en eng eng Living Individual
201 en.ner.roberta_large_finetuned_abbr roberta_ner_roberta_large_finetuned_abbr Named Entity Recognition English RoBertaForTokenClassification en eng eng Living Individual
202 en.ner.roberta_classics_ner roberta_ner_roberta_classics_ner Named Entity Recognition English RoBertaForTokenClassification en eng eng Living Individual
203 en.pos.roberta_base_english_upos roberta_pos_roberta_base_english_upos Part of Speech Tagging English RoBertaForTokenClassification en eng eng Living Individual
204 en.ner.roberta_large_ner_english roberta_ner_roberta_large_ner_english Named Entity Recognition English RoBertaForTokenClassification en eng eng Living Individual
205 en.ner.ner_gene_dna_rna_jnlpba_pubmed roberta_ner_ner_gene_dna_rna_jnlpba_pubmed Named Entity Recognition English RoBertaForTokenClassification en eng eng Living Individual
206 en.ner.ner_disease_ncbi_bionlp_bc5cdr_pubmed roberta_ner_ner_disease_ncbi_bionlp_bc5cdr_pubmed Named Entity Recognition English RoBertaForTokenClassification en eng eng Living Individual
207 myv.embed.w2v_cc_300d w2v_cc_300d Embeddings Erzya WordEmbeddingsModel nan myv myv Living Individual
208 fo.pos pos_farpahc Part of Speech Tagging Faroese PerceptronModel fo fao fao Living Individual
209 fi.embed.w2v_cc_300d w2v_cc_300d Embeddings Finnish WordEmbeddingsModel fi fin fin Living Individual
210 fi.pos pos_tdt Part of Speech Tagging Finnish PerceptronModel fi fin fin Living Individual
211 fi.lemma lemma_tdt Lemmatization Finnish LemmatizerModel fi fin fin Living Individual
212 fi.stopwords stopwords_iso Stop Words Removal Finnish StopWordsCleaner fi fin fin Living Individual
213 fi.lemma lemma_tdt Lemmatization Finnish LemmatizerModel fi fin fin Living Individual
214 fr.embed.camembert_large camembert_large Embeddings French CamemBertEmbeddings fr 639-2/T: fra639-2/B: fre fra Living Individual
215 fr.embed.camembert_base camembert_base Embeddings French CamemBertEmbeddings fr 639-2/T: fra639-2/B: fre fra Living Individual
216 fr.embed.camembert_ccnet4g camembert_base_ccnet_4gb Embeddings French CamemBertEmbeddings fr 639-2/T: fra639-2/B: fre fra Living Individual
217 fr.embed.camembert_base_ccnet camembert_base_ccnet Embeddings French CamemBertEmbeddings fr 639-2/T: fra639-2/B: fre fra Living Individual
218 fr.embed.camembert_oscar_4g camembert_base_oscar_4gb Embeddings French CamemBertEmbeddings fr 639-2/T: fra639-2/B: fre fra Living Individual
219 fr.embed.camembert_wiki_4g camembert_base_wikipedia_4gb Embeddings French CamemBertEmbeddings fr 639-2/T: fra639-2/B: fre fra Living Individual
220 fr.embed.albert albert_embeddings_fralbert_base Embeddings French AlbertEmbeddings fr 639-2/T: fra639-2/B: fre fra Living Individual
221 fr.embed.distilbert distilbert_embeddings_distilbert_base_fr_cased Embeddings French DistilBertEmbeddings fr 639-2/T: fra639-2/B: fre fra Living Individual
222 fr.embed.bert_base_fr_cased bert_embeddings_bert_base_fr_cased Embeddings French BertEmbeddings fr 639-2/T: fra639-2/B: fre fra Living Individual
223 fr.pos pos_sequoia Part of Speech Tagging French PerceptronModel fr 639-2/T: fra639-2/B: fre fra Living Individual
224 fr.pos pos_sequoia Part of Speech Tagging French PerceptronModel fr 639-2/T: fra639-2/B: fre fra Living Individual
225 fr.embed.french_roberta roberta_embeddings_french_roberta Embeddings French RoBertaEmbeddings fr 639-2/T: fra639-2/B: fre fra Living Individual
226 fr.lemma lemma_ftb Lemmatization French LemmatizerModel fr 639-2/T: fra639-2/B: fre fra Living Individual
227 fr.lemma lemma_ftb Lemmatization French LemmatizerModel fr 639-2/T: fra639-2/B: fre fra Living Individual
228 fr.stopwords stopwords_iso Stop Words Removal French StopWordsCleaner fr 639-2/T: fra639-2/B: fre fra Living Individual
229 fr.embed.roberta_base_wechsel_french roberta_embeddings_roberta_base_wechsel_french Embeddings French RoBertaEmbeddings fr 639-2/T: fra639-2/B: fre fra Living Individual
230 gd.embed.w2v_cc_300d w2v_cc_300d Embeddings Gaelic, Scottish Gaelic WordEmbeddingsModel gd gla gla Living Individual
231 gl.embed.w2v_cc_300d w2v_cc_300d Embeddings Galician WordEmbeddingsModel gl glg glg Living Individual
232 gl.lemma lemma_treegal Lemmatization Galician LemmatizerModel gl glg glg Living Individual
233 ka.embed.w2v_cc_300d w2v_cc_300d Embeddings Georgian WordEmbeddingsModel ka 639-2/T: kat639-2/B: geo kat Living Individual
234 de.embed.distilbert_base_de_cased distilbert_embeddings_distilbert_base_de_cased Embeddings German DistilBertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
235 de.embed.distilbert_base_german_cased distilbert_embeddings_distilbert_base_german_cased Embeddings German DistilBertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
236 de.embed.albert_german_ner albert_embeddings_albert_german_ner Embeddings German AlbertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
237 de.embed.bert_base_historical_german_rw_cased bert_embeddings_bert_base_historical_german_rw_cased Embeddings German BertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
238 de.embed.gbert_base bert_embeddings_gbert_base Embeddings German BertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
239 de.embed.german_financial_statements_bert bert_embeddings_german_financial_statements_bert Embeddings German BertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
240 de.stopwords stopwords_iso Stop Words Removal German StopWordsCleaner de 639-2/T: deu639-2/B: ger deu Living Individual
241 de.lemma lemma_spacylookup Lemmatization German LemmatizerModel de 639-2/T: deu639-2/B: ger deu Living Individual
242 de.embed.bert_base_german_dbmdz_uncased bert_embeddings_bert_base_german_dbmdz_uncased Embeddings German BertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
243 de.embed.roberta_base_wechsel_german roberta_embeddings_roberta_base_wechsel_german Embeddings German RoBertaEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
244 de.embed.gbert_large bert_embeddings_gbert_large Embeddings German BertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
245 de.embed.bert_base_5lang_cased bert_embeddings_bert_base_5lang_cased Embeddings German BertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
246 de.embed.bert_base_german_cased_oldvocab bert_embeddings_bert_base_german_cased_oldvocab Embeddings German BertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
247 de.embed.bert_base_de_cased bert_embeddings_bert_base_de_cased Embeddings German BertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
248 de.embed.bert_base_german_uncased bert_embeddings_bert_base_german_uncased Embeddings German BertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
249 de.embed.bert_base_german_dbmdz_cased bert_embeddings_bert_base_german_dbmdz_cased Embeddings German BertEmbeddings de 639-2/T: deu639-2/B: ger deu Living Individual
250 gom.embed.w2v_cc_300d w2v_cc_300d Embeddings Goan Konkani WordEmbeddingsModel nan nan gom Living Individual
251 gu.embed.RoBERTa_hindi_guj_san roberta_embeddings_RoBERTa_hindi_guj_san Embeddings Gujarati RoBertaEmbeddings gu guj guj Living Individual
252 gu.stopwords stopwords_iso Stop Words Removal Gujarati StopWordsCleaner gu guj guj Living Individual
253 he.stopwords stopwords_iso Stop Words Removal Hebrew StopWordsCleaner he heb heb Living Individual
254 hi.embed.distilbert_base_hi_cased distilbert_embeddings_distilbert_base_hi_cased Embeddings Hindi DistilBertEmbeddings hi hin hin Living Individual
255 hi.embed.indic_transformers_hi_distilbert distilbert_embeddings_indic_transformers_hi_distilbert Embeddings Hindi DistilBertEmbeddings hi hin hin Living Individual
256 hi.stopwords stopwords_iso Stop Words Removal Hindi StopWordsCleaner hi hin hin Living Individual
257 hi.embed.RoBERTa_hindi_guj_san roberta_embeddings_RoBERTa_hindi_guj_san Embeddings Hindi RoBertaEmbeddings hi hin hin Living Individual
258 hi.embed.indic_transformers_hi_roberta roberta_embeddings_indic_transformers_hi_roberta Embeddings Hindi RoBertaEmbeddings hi hin hin Living Individual
259 hi.embed.muril_adapted_local bert_embeddings_muril_adapted_local Embeddings Hindi BertEmbeddings hi hin hin Living Individual
260 hi.embed.indic_transformers_hi_bert bert_embeddings_indic_transformers_hi_bert Embeddings Hindi BertEmbeddings hi hin hin Living Individual
261 hu.lemma lemma_spacylookup Lemmatization Hungarian LemmatizerModel hu hun hun Living Individual
262 hu.stopwords stopwords_iso Stop Words Removal Hungarian StopWordsCleaner hu hun hun Living Individual
263 is.lemma lemma_icepahc Lemmatization Icelandic LemmatizerModel is 639-2/T: isl639-2/B: ice isl Living Individual
264 is.stopwords stopwords_iso Stop Words Removal Icelandic StopWordsCleaner is 639-2/T: isl639-2/B: ice isl Living Individual
265 id.embed.distilbert distilbert_embeddings_distilbert_base_indonesian Embeddings Indonesian DistilBertEmbeddings id ind ind Living Individual
266 id.pos pos_csui Part of Speech Tagging Indonesian PerceptronModel id ind ind Living Individual
267 id.embed.indo_roberta_small roberta_embeddings_indo_roberta_small Embeddings Indonesian RoBertaEmbeddings id ind ind Living Individual
268 id.embed.indonesian_roberta_base roberta_embeddings_indonesian_roberta_base Embeddings Indonesian RoBertaEmbeddings id ind ind Living Individual
269 id.pos.indonesian_roberta_base_posp_tagger roberta_pos_indonesian_roberta_base_posp_tagger Part of Speech Tagging Indonesian RoBertaForTokenClassification id ind ind Living Individual
270 id.lemma lemma_gsd Lemmatization Indonesian LemmatizerModel id ind ind Living Individual
271 id.lemma lemma_gsd Lemmatization Indonesian LemmatizerModel id ind ind Living Individual
272 id.embed.roberta_base_indonesian_522M roberta_embeddings_roberta_base_indonesian_522M Embeddings Indonesian RoBertaEmbeddings id ind ind Living Individual
273 id.stopwords stopwords_iso Stop Words Removal Indonesian StopWordsCleaner id ind ind Living Individual
274 id.embed.indonesian_roberta_large roberta_embeddings_indonesian_roberta_large Embeddings Indonesian RoBertaEmbeddings id ind ind Living Individual
275 ga.pos pos_idt Part of Speech Tagging Irish PerceptronModel ga gle gle Living Individual
276 ga.stopwords stopwords_iso Stop Words Removal Irish StopWordsCleaner ga gle gle Living Individual
277 it.embed.distilbert_base_it_cased distilbert_embeddings_distilbert_base_it_cased Embeddings Italian DistilBertEmbeddings it ita ita Living Individual
278 it.embed.BERTino distilbert_embeddings_BERTino Embeddings Italian DistilBertEmbeddings it ita ita Living Individual
279 it.stopwords stopwords_iso Stop Words Removal Italian StopWordsCleaner it ita ita Living Individual
280 it.pos pos_partut Part of Speech Tagging Italian PerceptronModel it ita ita Living Individual
281 it.embed.bert_base_italian_xxl_cased bert_embeddings_bert_base_italian_xxl_cased Embeddings Italian BertEmbeddings it ita ita Living Individual
282 it.embed.bert_base_italian_xxl_uncased bert_embeddings_bert_base_italian_xxl_uncased Embeddings Italian BertEmbeddings it ita ita Living Individual
283 it.embed.chefberto_italian_cased bert_embeddings_chefberto_italian_cased Embeddings Italian BertEmbeddings it ita ita Living Individual
284 it.embed.hseBert_it_cased bert_embeddings_hseBert_it_cased Embeddings Italian BertEmbeddings it ita ita Living Individual
285 it.embed.wineberto_italian_cased bert_embeddings_wineberto_italian_cased Embeddings Italian BertEmbeddings it ita ita Living Individual
286 it.pos pos_partut Part of Speech Tagging Italian PerceptronModel it ita ita Living Individual
287 it.lemma lemma_twittiro Lemmatization Italian LemmatizerModel it ita ita Living Individual
288 it.lemma lemma_twittiro Lemmatization Italian LemmatizerModel it ita ita Living Individual
289 it.lemma lemma_twittiro Lemmatization Italian LemmatizerModel it ita ita Living Individual
290 ja.embed.distilbert_base_ja_cased distilbert_embeddings_distilbert_base_ja_cased Embeddings Japanese DistilBertEmbeddings ja jpn jpn Living Individual
291 ja.embed.albert_base_japanese_v1 albert_embeddings_albert_base_japanese_v1 Embeddings Japanese AlbertEmbeddings ja jpn jpn Living Individual
292 ja.embed.bert_base_ja_cased bert_embeddings_bert_base_ja_cased Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
293 ja.embed.bert_base_japanese_char_v2 bert_embeddings_bert_base_japanese_char_v2 Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
294 ja.embed.bert_base_japanese_char_extended bert_embeddings_bert_base_japanese_char_extended Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
295 ja.embed.bert_large_japanese_char bert_embeddings_bert_large_japanese_char Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
296 ja.embed.bert_large_japanese bert_embeddings_bert_large_japanese Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
297 ja.embed.bert_small_japanese bert_embeddings_bert_small_japanese Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
298 ja.embed.bert_large_japanese_char_extended bert_embeddings_bert_large_japanese_char_extended Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
299 ja.pos pos_gsd Part of Speech Tagging Japanese PerceptronModel ja jpn jpn Living Individual
300 ja.embed.bert_small_japanese_fin bert_embeddings_bert_small_japanese_fin Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
301 ja.embed.bert_base_japanese_basic_char_v2 bert_embeddings_bert_base_japanese_basic_char_v2 Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
302 ja.stopwords stopwords_iso Stop Words Removal Japanese StopWordsCleaner ja jpn jpn Living Individual
303 ja.embed.bert_base_japanese_char_whole_word_masking bert_embeddings_bert_base_japanese_char_whole_word_masking Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
304 ja.embed.bert_base_japanese_char bert_embeddings_bert_base_japanese_char Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
305 ja.embed.bert_base_japanese_whole_word_masking bert_embeddings_bert_base_japanese_whole_word_masking Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
306 ja.embed.bert_base_japanese_v2 bert_embeddings_bert_base_japanese_v2 Embeddings Japanese BertEmbeddings ja jpn jpn Living Individual
307 jv.embed.distilbert distilbert_embeddings_javanese_distilbert_small Embeddings Javanese DistilBertEmbeddings jv jav jav Living Individual
308 jv.embed.javanese_distilbert_small_imdb distilbert_embeddings_javanese_distilbert_small_imdb Embeddings Javanese DistilBertEmbeddings jv jav jav Living Individual
309 jv.embed.javanese_roberta_small roberta_embeddings_javanese_roberta_small Embeddings Javanese RoBertaEmbeddings jv jav jav Living Individual
310 jv.embed.javanese_roberta_small_imdb roberta_embeddings_javanese_roberta_small_imdb Embeddings Javanese RoBertaEmbeddings jv jav jav Living Individual
311 jv.embed.javanese_bert_small_imdb bert_embeddings_javanese_bert_small_imdb Embeddings Javanese BertEmbeddings jv jav jav Living Individual
312 jv.embed.javanese_bert_small bert_embeddings_javanese_bert_small Embeddings Javanese BertEmbeddings jv jav jav Living Individual
313 kn.embed.KNUBert roberta_embeddings_KNUBert Embeddings Kannada RoBertaEmbeddings kn kan kan Living Individual
314 kn.embed.KanBERTo roberta_embeddings_KanBERTo Embeddings Kannada RoBertaEmbeddings kn kan kan Living Individual
315 kn.stopwords stopwords_iso Stop Words Removal Kannada StopWordsCleaner kn kan kan Living Individual
316 ky.stopwords stopwords_iso Stop Words Removal Kirghiz, Kyrgyz StopWordsCleaner ky kir kir Living Individual
317 ko.lemma lemma_gsd Lemmatization Korean LemmatizerModel ko kor kor Living Individual
318 ko.stopwords stopwords_iso Stop Words Removal Korean StopWordsCleaner ko kor kor Living Individual
319 ko.embed.roberta_ko_small roberta_embeddings_roberta_ko_small Embeddings Korean RoBertaEmbeddings ko kor kor Living Individual
320 ko.pos pos_gsd Part of Speech Tagging Korean PerceptronModel ko kor kor Living Individual
321 ko.embed.bert_kor_base bert_embeddings_bert_kor_base Embeddings Korean BertEmbeddings ko kor kor Living Individual
322 ko.embed.dbert bert_embeddings_dbert Embeddings Korean BertEmbeddings ko kor kor Living Individual
323 ko.embed.KR_FinBert bert_embeddings_KR_FinBert Embeddings Korean BertEmbeddings ko kor kor Living Individual
324 ko.embed.bert_base_v1_sports bert_embeddings_bert_base_v1_sports Embeddings Korean BertEmbeddings ko kor kor Living Individual
325 ko.lemma lemma_gsd Lemmatization Korean LemmatizerModel ko kor kor Living Individual
326 lb.stopwords stopwords_iso Stop Words Removal Letzeburgesch, Luxembourgish StopWordsCleaner lb ltz ltz Living Individual
327 lb.lemma lemma_spacylookup Lemmatization Letzeburgesch, Luxembourgish LemmatizerModel lb ltz ltz Living Individual
328 lb.embed.w2v_cc_300d w2v_cc_300d Embeddings Letzeburgesch, Luxembourgish WordEmbeddingsModel lb ltz ltz Living Individual
329 lij.stopwords stopwords_iso Stop Words Removal Ligurian StopWordsCleaner nan nan lij Living Individual
330 lt.embed.w2v_cc_300d w2v_cc_300d Embeddings Lithuanian WordEmbeddingsModel lt lit lit Living Individual
331 lt.lemma lemma_spacylookup Lemmatization Lithuanian LemmatizerModel lt lit lit Living Individual
332 lt.stopwords stopwords_iso Stop Words Removal Lithuanian StopWordsCleaner lt lit lit Living Individual
333 lmo.embed.w2v_cc_300d w2v_cc_300d Embeddings Lombard WordEmbeddingsModel nan nan lmo Living Individual
334 nds.embed.w2v_cc_300d w2v_cc_300d Embeddings Low German, Low Saxon WordEmbeddingsModel nan nds nds Living Individual
335 mk.stopwords stopwords_iso Stop Words Removal Macedonian StopWordsCleaner mk 639-2/T: mkd639-2/B: mac mkd Living Individual
336 mk.lemma lemma_spacylookup Lemmatization Macedonian LemmatizerModel mk 639-2/T: mkd639-2/B: mac mkd Living Individual
337 mk.embed.w2v_cc_300d w2v_cc_300d Embeddings Macedonian WordEmbeddingsModel mk 639-2/T: mkd639-2/B: mac mkd Living Individual
338 mai.embed.w2v_cc_300d w2v_cc_300d Embeddings Maithili WordEmbeddingsModel nan mai mai Living Individual
339 ml.stopwords stopwords_iso Stop Words Removal Malayalam StopWordsCleaner ml mal mal Living Individual
340 ml.embed.w2v_cc_300d w2v_cc_300d Embeddings Malayalam WordEmbeddingsModel ml mal mal Living Individual
341 mt.lemma lemma_mudt Lemmatization Maltese LemmatizerModel mt mlt mlt Living Individual
342 mt.pos pos_mudt Part of Speech Tagging Maltese PerceptronModel mt mlt mlt Living Individual
343 mt.embed.w2v_cc_300d w2v_cc_300d Embeddings Maltese WordEmbeddingsModel mt mlt mlt Living Individual
344 gv.embed.w2v_cc_300d w2v_cc_300d Embeddings Manx WordEmbeddingsModel gv glv glv Living Individual
345 mr.embed.distilbert distilbert_embeddings_marathi_distilbert Embeddings Marathi DistilBertEmbeddings mr mar mar Living Individual
346 mr.embed.albert albert_embeddings_marathi_albert Embeddings Marathi AlbertEmbeddings mr mar mar Living Individual
347 mr.embed.albert albert_embeddings_marathi_albert Embeddings Marathi AlbertEmbeddings mr mar mar Living Individual
348 mr.embed.albert_v2 albert_embeddings_marathi_albert_v2 Embeddings Marathi AlbertEmbeddings mr mar mar Living Individual
349 mr.embed.albert_v2 albert_embeddings_marathi_albert_v2 Embeddings Marathi AlbertEmbeddings mr mar mar Living Individual
350 mr.lemma lemma_ufal Lemmatization Marathi LemmatizerModel mr mar mar Living Individual
351 mr.stopwords stopwords_iso Stop Words Removal Marathi StopWordsCleaner mr mar mar Living Individual
352 mr.embed.marathi_bert bert_embeddings_marathi_bert Embeddings Marathi BertEmbeddings mr mar mar Living Individual
353 mr.embed.muril_adapted_local bert_embeddings_muril_adapted_local Embeddings Marathi BertEmbeddings mr mar mar Living Individual
354 mr.pos pos_ufal Part of Speech Tagging Marathi PerceptronModel mr mar mar Living Individual
355 mzn.embed.w2v_cc_300d w2v_cc_300d Embeddings Mazanderani WordEmbeddingsModel nan nan mzn Living Individual
356 min.embed.w2v_cc_300d w2v_cc_300d Embeddings Minangkabau WordEmbeddingsModel nan min min Living Individual
357 xmf.embed.w2v_cc_300d w2v_cc_300d Embeddings Mingrelian WordEmbeddingsModel nan nan xmf Living Individual
358 mwl.embed.w2v_cc_300d w2v_cc_300d Embeddings Mirandese WordEmbeddingsModel nan mwl mwl Living Individual
359 el.stopwords stopwords_iso Stop Words Removal Modern Greek (1453-) StopWordsCleaner el 639-2/T: ell639-2/B: gre ell Living Individual
360 ro.embed.distilbert_base_cased distilbert_embeddings_distilbert_base_ro_cased Embeddings Moldavian, Moldovan, Romanian DistilBertEmbeddings ro 639-2/T: ron639-2/B: rum ron Living Individual
361 ro.embed.ALR_BERT albert_embeddings_ALR_BERT Embeddings Moldavian, Moldovan, Romanian AlbertEmbeddings ro 639-2/T: ron639-2/B: rum ron Living Individual
362 ro.embed.w2v_cc_300d w2v_cc_300d Embeddings Moldavian, Moldovan, Romanian WordEmbeddingsModel ro 639-2/T: ron639-2/B: rum ron Living Individual
363 ro.stopwords stopwords_iso Stop Words Removal Moldavian, Moldovan, Romanian StopWordsCleaner ro 639-2/T: ron639-2/B: rum ron Living Individual
364 ro.pos pos_nonstandard Part of Speech Tagging Moldavian, Moldovan, Romanian PerceptronModel ro 639-2/T: ron639-2/B: rum ron Living Individual
365 ro.lemma lemma_spacylookup Lemmatization Moldavian, Moldovan, Romanian LemmatizerModel ro 639-2/T: ron639-2/B: rum ron Living Individual
366 nap.embed.w2v_cc_300d w2v_cc_300d Embeddings Neapolitan WordEmbeddingsModel nan nap nap Living Individual
367 new.embed.w2v_cc_300d w2v_cc_300d Embeddings Nepal Bhasa, Newari WordEmbeddingsModel nan new new Living Individual
368 frr.embed.w2v_cc_300d w2v_cc_300d Embeddings Northern Frisian WordEmbeddingsModel nan frr frr Living Individual
369 sme.lemma lemma_giella Lemmatization Northern Sami LemmatizerModel se sme sme Living Individual
370 sme.pos pos_giella Part of Speech Tagging Northern Sami PerceptronModel se sme sme Living Individual
371 nso.embed.w2v_cc_300d w2v_cc_300d Embeddings Northern Sotho, Pedi, Sepedi WordEmbeddingsModel nan nso nso Living Individual
372 nb.stopwords stopwords_iso Stop Words Removal Norwegian Bokmål StopWordsCleaner nb nob nob Living Individual
373 nb.lemma lemma_spacylookup Lemmatization Norwegian Bokmål LemmatizerModel nb nob nob Living Individual
374 nn.embed.w2v_cc_300d w2v_cc_300d Embeddings Norwegian Nynorsk WordEmbeddingsModel nn nno nno Living Individual
375 oc.embed.w2v_cc_300d w2v_cc_300d Embeddings Occitan (post 1500) WordEmbeddingsModel oc oci oci Living Individual
376 os.embed.w2v_cc_300d w2v_cc_300d Embeddings Ossetian, Ossetic WordEmbeddingsModel os oss oss Living Individual
377 pa.embed.w2v_cc_300d w2v_cc_300d Embeddings Panjabi, Punjabi WordEmbeddingsModel pa pan pan Living Individual
378 pa.embed.muril_adapted_local bert_embeddings_muril_adapted_local Embeddings Panjabi, Punjabi BertEmbeddings pa pan pan Living Individual
379 pfl.embed.w2v_cc_300d w2v_cc_300d Embeddings Pfaelzisch WordEmbeddingsModel nan nan pfl Living Individual
380 pms.embed.w2v_cc_300d w2v_cc_300d Embeddings Piemontese WordEmbeddingsModel nan nan pms Living Individual
381 pl.embed.distilbert_base_cased distilbert_embeddings_distilbert_base_pl_cased Embeddings Polish DistilBertEmbeddings pl pol pol Living Individual
382 pl.stopwords stopwords_iso Stop Words Removal Polish StopWordsCleaner pl pol pol Living Individual
383 pl.embed.w2v_cc_300d w2v_cc_300d Embeddings Polish WordEmbeddingsModel pl pol pol Living Individual
384 pl.lemma lemma_lfg Lemmatization Polish LemmatizerModel pl pol pol Living Individual
385 pt.med_ner.deid.subentity ner_deid_subentity De-identification Portuguese MedicalNerModel pt por por Living Individual
386 pt.med_ner.deid.generic ner_deid_generic De-identification Portuguese MedicalNerModel pt por por Living Individual
387 pt.med_ner.deid ner_deid_generic De-identification Portuguese MedicalNerModel pt por por Living Individual
388 pt.embed.distilbert_base_cased distilbert_embeddings_distilbert_base_pt_cased Embeddings Portuguese DistilBertEmbeddings pt por por Living Individual
389 pt.embed.BR_BERTo roberta_embeddings_BR_BERTo Embeddings Portuguese RoBertaEmbeddings pt por por Living Individual
390 pt.embed.gs_all biobert_embeddings_all Embeddings Portuguese BertEmbeddings pt por por Living Individual
391 pt.stopwords stopwords_iso Stop Words Removal Portuguese StopWordsCleaner pt por por Living Individual
392 pt.embed.gs_clinical biobert_embeddings_clinical Embeddings Portuguese BertEmbeddings pt por por Living Individual
393 pt.embed.gs_biomedical biobert_embeddings_biomedical Embeddings Portuguese BertEmbeddings pt por por Living Individual
394 pt.lemma lemma_bosque Lemmatization Portuguese LemmatizerModel pt por por Living Individual
395 pt.lemma lemma_bosque Lemmatization Portuguese LemmatizerModel pt por por Living Individual
396 pt.embed.bert_base_portuguese_cased_finetuned_tcu_acordaos bert_embeddings_bert_base_portuguese_cased_finetuned_tcu_acordaos Embeddings Portuguese BertEmbeddings pt por por Living Individual
397 pt.ner.satellite_instrument_roberta_NER roberta_ner_satellite_instrument_roberta_NER Named Entity Recognition Portuguese RoBertaForTokenClassification pt por por Living Individual
398 pt.embed.bert_small_gl_cased bert_embeddings_bert_small_gl_cased Embeddings Portuguese BertEmbeddings pt por por Living Individual
399 pt.embed.bert_large_cased_pt_lenerbr bert_embeddings_bert_large_cased_pt_lenerbr Embeddings Portuguese BertEmbeddings pt por por Living Individual
400 pt.embed.bert_large_portuguese_cased bert_embeddings_bert_large_portuguese_cased Embeddings Portuguese BertEmbeddings pt por por Living Individual
401 pt.embed.bert_base_cased_pt_lenerbr bert_embeddings_bert_base_cased_pt_lenerbr Embeddings Portuguese BertEmbeddings pt por por Living Individual
402 pt.embed.bert_base_portuguese_cased_finetuned_peticoes bert_embeddings_bert_base_portuguese_cased_finetuned_peticoes Embeddings Portuguese BertEmbeddings pt por por Living Individual
403 pt.embed.bert_base_portuguese_cased bert_embeddings_bert_base_portuguese_cased Embeddings Portuguese BertEmbeddings pt por por Living Individual
404 pt.embed.bert_base_pt_cased bert_embeddings_bert_base_pt_cased Embeddings Portuguese BertEmbeddings pt por por Living Individual
405 pt.embed.bert_base_gl_cased bert_embeddings_bert_base_gl_cased Embeddings Portuguese BertEmbeddings pt por por Living Individual
406 rm.embed.w2v_cc_300d w2v_cc_300d Embeddings Romansh WordEmbeddingsModel rm roh roh Living Individual
407 ru.embed.distilbert_base_cased distilbert_embeddings_distilbert_base_ru_cased Embeddings Russian DistilBertEmbeddings ru rus rus Living Individual
408 ru.pos pos_syntagrus Part of Speech Tagging Russian PerceptronModel ru rus rus Living Individual
409 ru.lemma lemma_gsd Lemmatization Russian LemmatizerModel ru rus rus Living Individual
410 ru.lemma lemma_gsd Lemmatization Russian LemmatizerModel ru rus rus Living Individual
411 ru.embed.ruRoberta_large roberta_embeddings_ruRoberta_large Embeddings Russian RoBertaEmbeddings ru rus rus Living Individual
412 ru.pos pos_syntagrus Part of Speech Tagging Russian PerceptronModel ru rus rus Living Individual
413 ru.stopwords stopwords_iso Stop Words Removal Russian StopWordsCleaner ru rus rus Living Individual
414 ru.embed.roberta_base_russian_v0 roberta_embeddings_roberta_base_russian_v0 Embeddings Russian RoBertaEmbeddings ru rus rus Living Individual
415 ru.embed.bert_base_ru_cased bert_embeddings_bert_base_ru_cased Embeddings Russian BertEmbeddings ru rus rus Living Individual
416 ru.embed.w2v_cc_300d w2v_cc_300d Embeddings Russian WordEmbeddingsModel ru rus rus Living Individual
417 sco.embed.w2v_cc_300d w2v_cc_300d Embeddings Scots WordEmbeddingsModel nan sco sco Living Individual
418 sr.lemma lemma_spacylookup Lemmatization Serbian LemmatizerModel sr srp srp Living Individual
419 sr.embed.w2v_cc_300d w2v_cc_300d Embeddings Serbian WordEmbeddingsModel sr srp srp Living Individual
420 sr.lemma lemma_spacylookup Lemmatization Serbian LemmatizerModel sr srp srp Living Individual
421 sr.stopwords stopwords_iso Stop Words Removal Serbian StopWordsCleaner sr srp srp Living Individual
422 scn.embed.w2v_cc_300d w2v_cc_300d Embeddings Sicilian WordEmbeddingsModel nan scn scn Living Individual
423 sd.embed.w2v_cc_300d w2v_cc_300d Embeddings Sindhi WordEmbeddingsModel sd snd snd Living Individual
424 si.stopwords stopwords_iso Stop Words Removal Sinhala, Sinhalese StopWordsCleaner si sin sin Living Individual
425 si.embed.w2v_cc_300d w2v_cc_300d Embeddings Sinhala, Sinhalese WordEmbeddingsModel si sin sin Living Individual
426 sk.stopwords stopwords_iso Stop Words Removal Slovak StopWordsCleaner sk 639-2/T: slk639-2/B: slo slk Living Individual
427 sk.lemma lemma_snk Lemmatization Slovak LemmatizerModel sk 639-2/T: slk639-2/B: slo slk Living Individual
428 sk.embed.w2v_cc_300d w2v_cc_300d Embeddings Slovak WordEmbeddingsModel sk 639-2/T: slk639-2/B: slo slk Living Individual
429 sl.lemma lemma_sst Lemmatization Slovenian LemmatizerModel sl slv slv Living Individual
430 sl.stopwords stopwords_iso Stop Words Removal Slovenian StopWordsCleaner sl slv slv Living Individual
431 sl.pos pos_sst Part of Speech Tagging Slovenian PerceptronModel sl slv slv Living Individual
432 sl.embed.w2v_cc_300d w2v_cc_300d Embeddings Slovenian WordEmbeddingsModel sl slv slv Living Individual
433 so.embed.w2v_cc_300d w2v_cc_300d Embeddings Somali WordEmbeddingsModel so som som Living Individual
434 su.embed.w2v_cc_300d w2v_cc_300d Embeddings Sundanese WordEmbeddingsModel su sun sun Living Individual
435 su.embed.sundanese_roberta_base roberta_embeddings_sundanese_roberta_base Embeddings Sundanese RoBertaEmbeddings su sun sun Living Individual
436 sv.stopwords stopwords_iso Stop Words Removal Swedish StopWordsCleaner sv swe swe Living Individual
437 sv.embed.w2v_cc_300d w2v_cc_300d Embeddings Swedish WordEmbeddingsModel sv swe swe Living Individual
438 sv.lemma lemma_lines Lemmatization Swedish LemmatizerModel sv swe swe Living Individual
439 sv.lemma lemma_lines Lemmatization Swedish LemmatizerModel sv swe swe Living Individual
440 tl.lemma lemma_spacylookup Lemmatization Tagalog LemmatizerModel tl tgl tgl Living Individual
441 tl.embed.w2v_cc_300d w2v_cc_300d Embeddings Tagalog WordEmbeddingsModel tl tgl tgl Living Individual
442 tl.stopwords stopwords_iso Stop Words Removal Tagalog StopWordsCleaner tl tgl tgl Living Individual
443 tl.embed.roberta_tagalog_large roberta_embeddings_roberta_tagalog_large Embeddings Tagalog RoBertaEmbeddings tl tgl tgl Living Individual
444 tl.embed.roberta_tagalog_base roberta_embeddings_roberta_tagalog_base Embeddings Tagalog RoBertaEmbeddings tl tgl tgl Living Individual
445 tg.embed.w2v_cc_300d w2v_cc_300d Embeddings Tajik WordEmbeddingsModel tg tgk tgk Living Individual
446 ta.stopwords stopwords_iso Stop Words Removal Tamil StopWordsCleaner ta tam tam Living Individual
447 ta.embed.w2v_cc_300d w2v_cc_300d Embeddings Tamil WordEmbeddingsModel ta tam tam Living Individual
448 ta.embed.muril_adapted_local bert_embeddings_muril_adapted_local Embeddings Tamil BertEmbeddings ta tam tam Living Individual
449 tt.stopwords stopwords_iso Stop Words Removal Tatar StopWordsCleaner tt tat tat Living Individual
450 tt.embed.w2v_cc_300d w2v_cc_300d Embeddings Tatar WordEmbeddingsModel tt tat tat Living Individual
451 te.embed.indic_transformers_te_bert bert_embeddings_indic_transformers_te_bert Embeddings Telugu BertEmbeddings te tel tel Living Individual
452 te.embed.telugu_bertu bert_embeddings_telugu_bertu Embeddings Telugu BertEmbeddings te tel tel Living Individual
453 te.embed.muril_adapted_local bert_embeddings_muril_adapted_local Embeddings Telugu BertEmbeddings te tel tel Living Individual
454 te.embed.indic_transformers_te_roberta roberta_embeddings_indic_transformers_te_roberta Embeddings Telugu RoBertaEmbeddings te tel tel Living Individual
455 te.stopwords stopwords_iso Stop Words Removal Telugu StopWordsCleaner te tel tel Living Individual
456 te.lemma lemma_mtg Lemmatization Telugu LemmatizerModel te tel tel Living Individual
457 te.embed.w2v_cc_300d w2v_cc_300d Embeddings Telugu WordEmbeddingsModel te tel tel Living Individual
458 th.embed.distilbert_base_cased distilbert_embeddings_distilbert_base_th_cased Embeddings Thai DistilBertEmbeddings th tha tha Living Individual
459 th.stopwords stopwords_iso Stop Words Removal Thai StopWordsCleaner th tha tha Living Individual
460 th.embed.w2v_cc_300d w2v_cc_300d Embeddings Thai WordEmbeddingsModel th tha tha Living Individual
461 ti.stopwords stopwords_iso Stop Words Removal Tigrinya StopWordsCleaner ti tir tir Living Individual
462 als.embed.w2v_cc_300d w2v_cc_300d Embeddings Tosk Albanian WordEmbeddingsModel nan nan als Living Individual
463 tn.stopwords stopwords_iso Stop Words Removal Tswana StopWordsCleaner tn tsn tsn Living Individual
464 tr.embed.distilbert_base_cased distilbert_embeddings_distilbert_base_tr_cased Embeddings Turkish DistilBertEmbeddings tr tur tur Living Individual
465 tr.lemma lemma_penn Lemmatization Turkish LemmatizerModel tr tur tur Living Individual
466 tr.stopwords stopwords_iso Stop Words Removal Turkish StopWordsCleaner tr tur tur Living Individual
467 tr.lemma lemma_penn Lemmatization Turkish LemmatizerModel tr tur tur Living Individual
468 tr.pos pos_boun Part of Speech Tagging Turkish PerceptronModel tr tur tur Living Individual
469 tr.embed.w2v_cc_300d w2v_cc_300d Embeddings Turkish WordEmbeddingsModel tr tur tur Living Individual
470 tr.lemma lemma_penn Lemmatization Turkish LemmatizerModel tr tur tur Living Individual
471 tr.pos pos_boun Part of Speech Tagging Turkish PerceptronModel tr tur tur Living Individual
472 tr.pos pos_boun Part of Speech Tagging Turkish PerceptronModel tr tur tur Living Individual
473 tr.lemma lemma_penn Lemmatization Turkish LemmatizerModel tr tur tur Living Individual
474 tr.lemma lemma_penn Lemmatization Turkish LemmatizerModel tr tur tur Living Individual
475 tk.embed.w2v_cc_300d w2v_cc_300d Embeddings Turkmen WordEmbeddingsModel tk tuk tuk Living Individual
476 ug.embed.w2v_cc_300d w2v_cc_300d Embeddings Uighur, Uyghur WordEmbeddingsModel ug uig uig Living Individual
477 uk.embed.distilbert_base_cased distilbert_embeddings_distilbert_base_uk_cased Embeddings Ukrainian DistilBertEmbeddings uk ukr ukr Living Individual
478 uk.embed.ukr_roberta_base roberta_embeddings_ukr_roberta_base Embeddings Ukrainian RoBertaEmbeddings uk ukr ukr Living Individual
479 uk.stopwords stopwords_iso Stop Words Removal Ukrainian StopWordsCleaner uk ukr ukr Living Individual
480 uk.embed.w2v_cc_300d w2v_cc_300d Embeddings Ukrainian WordEmbeddingsModel uk ukr ukr Living Individual
481 uk.pos.bert_large_slavic_cyrillic_upos bert_pos_bert_large_slavic_cyrillic_upos Part of Speech Tagging Ukrainian BertForTokenClassification uk ukr ukr Living Individual
482 uk.pos.bert_base_slavic_cyrillic_upos bert_pos_bert_base_slavic_cyrillic_upos Part of Speech Tagging Ukrainian BertForTokenClassification uk ukr ukr Living Individual
483 hsb.embed.w2v_cc_300d w2v_cc_300d Embeddings Upper Sorbian WordEmbeddingsModel nan hsb hsb Living Individual
484 ur.embed.distilbert_base_cased distilbert_embeddings_distilbert_base_ur_cased Embeddings Urdu DistilBertEmbeddings ur urd urd Living Individual
485 ur.embed.muril_adapted_local bert_embeddings_muril_adapted_local Embeddings Urdu BertEmbeddings ur urd urd Living Individual
486 ur.embed.roberta_urdu_small roberta_embeddings_roberta_urdu_small Embeddings Urdu RoBertaEmbeddings ur urd urd Living Individual
487 ur.lemma lemma_udtb Lemmatization Urdu LemmatizerModel ur urd urd Living Individual
488 ur.lemma lemma_udtb Lemmatization Urdu LemmatizerModel ur urd urd Living Individual
489 ur.pos pos_udtb Part of Speech Tagging Urdu PerceptronModel ur urd urd Living Individual
490 ur.embed.w2v_cc_300d w2v_cc_300d Embeddings Urdu WordEmbeddingsModel ur urd urd Living Individual
491 ur.stopwords stopwords_iso Stop Words Removal Urdu StopWordsCleaner ur urd urd Living Individual
492 vec.embed.w2v_cc_300d w2v_cc_300d Embeddings Venetian WordEmbeddingsModel nan nan vec Living Individual
493 vi.stopwords stopwords_iso Stop Words Removal Vietnamese StopWordsCleaner vi vie vie Living Individual
494 vi.embed.w2v_cc_300d w2v_cc_300d Embeddings Vietnamese WordEmbeddingsModel vi vie vie Living Individual
495 vls.embed.w2v_cc_300d w2v_cc_300d Embeddings Vlaams WordEmbeddingsModel nan nan vls Living Individual
496 wa.embed.w2v_cc_300d w2v_cc_300d Embeddings Walloon WordEmbeddingsModel wa wln wln Living Individual
497 war.embed.w2v_cc_300d w2v_cc_300d Embeddings Waray (Philippines) WordEmbeddingsModel nan war war Living Individual
498 hyw.pos pos_armtdp Part of Speech Tagging Western Armenian PerceptronModel nan nan hyw Living Individual
499 hyw.lemma lemma_armtdp Lemmatization Western Armenian LemmatizerModel nan nan hyw Living Individual
500 fy.embed.w2v_cc_300d w2v_cc_300d Embeddings Western Frisian WordEmbeddingsModel fy fry fry Living Individual
501 pnb.embed.w2v_cc_300d w2v_cc_300d Embeddings Western Panjabi WordEmbeddingsModel nan nan pnb Living Individual
502 wo.pos pos_wtb Part of Speech Tagging Wolof PerceptronModel wo wol wol Living Individual
503 sah.embed.w2v_cc_300d w2v_cc_300d Embeddings Yakut WordEmbeddingsModel nan sah sah Living Individual
504 yo.embed.w2v_cc_300d w2v_cc_300d Embeddings Yoruba WordEmbeddingsModel yo yor yor Living Individual
505 zea.embed.w2v_cc_300d w2v_cc_300d Embeddings Zeeuws WordEmbeddingsModel nan nan zea Living Individual
506 sq.stopwords stopwords_iso Stop Words Removal Albanian StopWordsCleaner sq 639-2/T: sqi639-2/B: alb sqi Living Macrolanguage
507 sq.embed.w2v_cc_300d w2v_cc_300d Embeddings Albanian WordEmbeddingsModel sq 639-2/T: sqi639-2/B: alb sqi Living Macrolanguage
508 ar.embed.distilbert distilbert_embeddings_distilbert_base_ar_cased Embeddings Arabic DistilBertEmbeddings ar ara ara Living Macrolanguage
509 ar.embed.albert albert_embeddings_albert_base_arabic Embeddings Arabic AlbertEmbeddings ar ara ara Living Macrolanguage
510 ar.embed.albert_xlarge_arabic albert_embeddings_albert_xlarge_arabic Embeddings Arabic AlbertEmbeddings ar ara ara Living Macrolanguage
511 ar.embed.albert_large_arabic albert_embeddings_albert_large_arabic Embeddings Arabic AlbertEmbeddings ar ara ara Living Macrolanguage
512 ar.pos.arabic_camelbert_msa_pos_msa bert_pos_bert_base_arabic_camelbert_msa_pos_msa Part of Speech Tagging Arabic BertForTokenClassification ar ara ara Living Macrolanguage
513 ar.pos.arabic_camelbert_mix_pos_egy bert_pos_bert_base_arabic_camelbert_mix_pos_egy Part of Speech Tagging Arabic BertForTokenClassification ar ara ara Living Macrolanguage
514 ar.pos.arabic_camelbert_da_pos_glf bert_pos_bert_base_arabic_camelbert_da_pos_glf Part of Speech Tagging Arabic BertForTokenClassification ar ara ara Living Macrolanguage
515 ar.pos.arabic_camelbert_ca_pos_glf bert_pos_bert_base_arabic_camelbert_ca_pos_glf Part of Speech Tagging Arabic BertForTokenClassification ar ara ara Living Macrolanguage
516 ar.pos.arabic_camelbert_msa_pos_egy bert_pos_bert_base_arabic_camelbert_msa_pos_egy Part of Speech Tagging Arabic BertForTokenClassification ar ara ara Living Macrolanguage
517 ar.pos.arabic_camelbert_ca_pos_egy bert_pos_bert_base_arabic_camelbert_ca_pos_egy Part of Speech Tagging Arabic BertForTokenClassification ar ara ara Living Macrolanguage
518 ar.pos.arabic_camelbert_msa_pos_glf bert_pos_bert_base_arabic_camelbert_msa_pos_glf Part of Speech Tagging Arabic BertForTokenClassification ar ara ara Living Macrolanguage
519 ar.pos.arabic_camelbert_mix_pos_glf bert_pos_bert_base_arabic_camelbert_mix_pos_glf Part of Speech Tagging Arabic BertForTokenClassification ar ara ara Living Macrolanguage
520 ar.pos.arabic_camelbert_da_pos_egy bert_pos_bert_base_arabic_camelbert_da_pos_egy Part of Speech Tagging Arabic BertForTokenClassification ar ara ara Living Macrolanguage
521 ar.stopwords stopwords_iso Stop Words Removal Arabic StopWordsCleaner ar ara ara Living Macrolanguage
522 ar.embed.multi_dialect_bert_base_arabic bert_embeddings_multi_dialect_bert_base_arabic Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
523 ar.ner.arabic_camelbert_da_ner bert_ner_bert_base_arabic_camelbert_da_ner Named Entity Recognition Arabic BertForTokenClassification ar ara ara Living Macrolanguage
524 ar.ner.arabic_camelbert_mix_ner bert_ner_bert_base_arabic_camelbert_mix_ner Named Entity Recognition Arabic BertForTokenClassification ar ara ara Living Macrolanguage
525 ar.pos pos_padt Part of Speech Tagging Arabic PerceptronModel ar ara ara Living Macrolanguage
526 ar.ner.multilingual_cased_ner_hrl bert_ner_bert_base_multilingual_cased_ner_hrl Named Entity Recognition Arabic BertForTokenClassification ar ara ara Living Macrolanguage
527 ar.ner.arabic_camelbert_msa_ner bert_ner_bert_base_arabic_camelbert_msa_ner Named Entity Recognition Arabic BertForTokenClassification ar ara ara Living Macrolanguage
528 ar.ner.ANER bert_ner_ANER Named Entity Recognition Arabic BertForTokenClassification ar ara ara Living Macrolanguage
529 ar.ner.arabert_ner bert_ner_arabert_ner Named Entity Recognition Arabic BertForTokenClassification ar ara ara Living Macrolanguage
530 ar.lemma lemma_padt Lemmatization Arabic LemmatizerModel ar ara ara Living Macrolanguage
531 ar.pos.arabic_camelbert_mix_pos_msa bert_pos_bert_base_arabic_camelbert_mix_pos_msa Part of Speech Tagging Arabic BertForTokenClassification ar ara ara Living Macrolanguage
532 ar.embed.mbert_ar_c19 bert_embeddings_mbert_ar_c19 Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
533 ar.embed.bert_base_arabic_camelbert_msa_half bert_embeddings_bert_base_arabic_camelbert_msa_half Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
534 ar.embed.bert_large_arabertv02 bert_embeddings_bert_large_arabertv02 Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
535 ar.embed.AraBertMo_base_V1 bert_embeddings_AraBertMo_base_V1 Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
536 ar.embed.DarijaBERT bert_embeddings_DarijaBERT Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
537 ar.embed.bert_base_arabertv02 bert_embeddings_bert_base_arabertv02 Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
538 ar.embed.arabert_c19 bert_embeddings_arabert_c19 Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
539 ar.embed.bert_base_arabic_camelbert_msa bert_embeddings_bert_base_arabic_camelbert_msa Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
540 ar.embed.bert_base_arabertv2 bert_embeddings_bert_base_arabertv2 Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
541 ar.embed.bert_base_arabic bert_embeddings_bert_base_arabic Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
542 ar.embed.Ara_DialectBERT bert_embeddings_Ara_DialectBERT Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
543 ar.embed.MARBERT bert_embeddings_MARBERT Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
544 ar.embed.bert_base_arabic_camelbert_msa_eighth bert_embeddings_bert_base_arabic_camelbert_msa_eighth Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
545 ar.embed.MARBERTv2 bert_embeddings_MARBERTv2 Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
546 ar.embed.bert_large_arabertv2 bert_embeddings_bert_large_arabertv2 Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
547 ar.embed.bert_base_arabert bert_embeddings_bert_base_arabert Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
548 ar.embed.bert_base_arabertv01 bert_embeddings_bert_base_arabertv01 Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
549 ar.embed.bert_mini_arabic bert_embeddings_bert_mini_arabic Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
550 ar.embed.bert_large_arabic bert_embeddings_bert_large_arabic Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
551 ar.embed.bert_large_arabertv02_twitter bert_embeddings_bert_large_arabertv02_twitter Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
552 ar.embed.dziribert bert_embeddings_dziribert Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
553 ar.embed.bert_base_arabertv02_twitter bert_embeddings_bert_base_arabertv02_twitter Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
554 ar.embed.bert_medium_arabic bert_embeddings_bert_medium_arabic Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
555 ar.pos.arabic_camelbert_da_pos_msa bert_pos_bert_base_arabic_camelbert_da_pos_msa Part of Speech Tagging Arabic BertForTokenClassification ar ara ara Living Macrolanguage
556 ar.embed.bert_base_qarib bert_embeddings_bert_base_qarib Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
557 ar.embed.bert_base_qarib60_860k bert_embeddings_bert_base_qarib60_860k Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
558 ar.embed.bert_base_qarib60_1790k bert_embeddings_bert_base_qarib60_1790k Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
559 ar.embed.bert_base_arabic_camelbert_msa_sixteenth bert_embeddings_bert_base_arabic_camelbert_msa_sixteenth Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
560 ar.embed.bert_base_arabic_camelbert_mix bert_embeddings_bert_base_arabic_camelbert_mix Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
561 ar.embed.bert_base_arabic_camelbert_msa_quarter bert_embeddings_bert_base_arabic_camelbert_msa_quarter Embeddings Arabic BertEmbeddings ar ara ara Living Macrolanguage
562 az.embed.w2v_cc_300d w2v_cc_300d Embeddings Azerbaijani WordEmbeddingsModel az aze aze Living Macrolanguage
563 az.stopwords stopwords_iso Stop Words Removal Azerbaijani StopWordsCleaner az aze aze Living Macrolanguage
564 zh.embed.distilbert_base_cased distilbert_embeddings_distilbert_base_zh_cased Embeddings Chinese DistilBertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
565 zh.embed.wobert_chinese_plus_base bert_embeddings_wobert_chinese_plus_base Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
566 zh.embed.bert_base_chinese_jinyong bert_embeddings_bert_base_chinese_jinyong Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
567 zh.embed.rbt3 bert_embeddings_rbt3 Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
568 zh.embed.jdt_fin_roberta_wwm bert_embeddings_jdt_fin_roberta_wwm Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
569 zh.embed.mengzi_oscar_base bert_embeddings_mengzi_oscar_base Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
570 zh.embed.roberta_base_wechsel_chinese roberta_embeddings_roberta_base_wechsel_chinese Embeddings Chinese RoBertaEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
571 zh.embed.sikubert bert_embeddings_sikubert Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
572 zh.embed.jdt_fin_roberta_wwm_large bert_embeddings_jdt_fin_roberta_wwm_large Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
573 zh.embed.rbtl3 bert_embeddings_rbtl3 Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
574 zh.embed.macbert4csc_base_chinese bert_embeddings_macbert4csc_base_chinese Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
575 zh.pos.chinese_roberta_large_upos bert_pos_chinese_roberta_large_upos Part of Speech Tagging Chinese BertForTokenClassification zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
576 zh.pos.chinese_roberta_base_upos bert_pos_chinese_roberta_base_upos Part of Speech Tagging Chinese BertForTokenClassification zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
577 zh.pos.chinese_bert_wwm_ext_upos bert_pos_chinese_bert_wwm_ext_upos Part of Speech Tagging Chinese BertForTokenClassification zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
578 zh.pos pos_gsdsimp Part of Speech Tagging Chinese PerceptronModel zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
579 zh.pos pos_gsdsimp Part of Speech Tagging Chinese PerceptronModel zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
580 zh.stopwords stopwords_iso Stop Words Removal Chinese StopWordsCleaner zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
581 zh.pos.bert_base_chinese_pos bert_pos_bert_base_chinese_pos Part of Speech Tagging Chinese BertForTokenClassification zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
582 zh.embed.rbt6 bert_embeddings_rbt6 Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
583 zh.embed.sikuroberta bert_embeddings_sikuroberta Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
584 zh.embed.uer_large bert_embeddings_uer_large Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
585 zh.embed.env_bert_chinese bert_embeddings_env_bert_chinese Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
586 zh.embed.chinese_roberta_wwm_ext bert_embeddings_chinese_roberta_wwm_ext Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
587 zh.embed.chinese_macbert_base bert_embeddings_chinese_macbert_base Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
588 zh.embed.bert_base_zh_cased bert_embeddings_bert_base_zh_cased Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
589 zh.embed.bert_large_chinese bert_embeddings_bert_large_chinese Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
590 zh.embed.chinese_roberta_wwm_large_ext_fix_mlm bert_embeddings_chinese_roberta_wwm_large_ext_fix_mlm Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
591 zh.embed.chinese_roberta_wwm_ext_large bert_embeddings_chinese_roberta_wwm_ext_large Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
592 zh.embed.chinese_bert_wwm_ext bert_embeddings_chinese_bert_wwm_ext Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
593 zh.embed.chinese_macbert_large bert_embeddings_chinese_macbert_large Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
594 zh.embed.mengzi_oscar_base_retrieval bert_embeddings_mengzi_oscar_base_retrieval Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
595 zh.embed.mengzi_bert_base_fin bert_embeddings_mengzi_bert_base_fin Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
596 zh.embed.wobert_chinese_base bert_embeddings_wobert_chinese_base Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
597 zh.embed.wobert_chinese_plus bert_embeddings_wobert_chinese_plus Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
598 zh.embed.rbt4 bert_embeddings_rbt4 Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
599 zh.embed.mengzi_oscar_base_caption bert_embeddings_mengzi_oscar_base_caption Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
600 zh.embed.mengzi_bert_base bert_embeddings_mengzi_bert_base Embeddings Chinese BertEmbeddings zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
601 zh.embed.w2v_cc_300d w2v_cc_300d Embeddings Chinese WordEmbeddingsModel zh 639-2/T: zho639-2/B: chi zho Living Macrolanguage
602 et.stopwords stopwords_iso Stop Words Removal Estonian StopWordsCleaner et est est Living Macrolanguage
603 et.pos pos_edt Part of Speech Tagging Estonian PerceptronModel et est est Living Macrolanguage
604 et.embed.w2v_cc_300d w2v_cc_300d Embeddings Estonian WordEmbeddingsModel et est est Living Macrolanguage
605 et.lemma lemma_ewt Lemmatization Estonian LemmatizerModel et est est Living Macrolanguage
606 et.lemma lemma_ewt Lemmatization Estonian LemmatizerModel et est est Living Macrolanguage
607 lv.stopwords stopwords_iso Stop Words Removal Latvian StopWordsCleaner lv lav lav Living Macrolanguage
608 lv.pos pos_lvtb Part of Speech Tagging Latvian PerceptronModel lv lav lav Living Macrolanguage
609 mg.embed.w2v_cc_300d w2v_cc_300d Embeddings Malagasy WordEmbeddingsModel mg mlg mlg Living Macrolanguage
610 ms.embed.albert albert_embeddings_albert_large_bahasa_cased Embeddings Malay (macrolanguage) AlbertEmbeddings ms 639-2/T: msa639-2/B: may msa Living Macrolanguage
611 ms.embed.distilbert distilbert_embeddings_malaysian_distilbert_small Embeddings Malay (macrolanguage) DistilBertEmbeddings ms 639-2/T: msa639-2/B: may msa Living Macrolanguage
612 ms.embed.albert_tiny_bahasa_cased albert_embeddings_albert_tiny_bahasa_cased Embeddings Malay (macrolanguage) AlbertEmbeddings ms 639-2/T: msa639-2/B: may msa Living Macrolanguage
613 ms.embed.albert_base_bahasa_cased albert_embeddings_albert_base_bahasa_cased Embeddings Malay (macrolanguage) AlbertEmbeddings ms 639-2/T: msa639-2/B: may msa Living Macrolanguage
614 ms.embed.w2v_cc_300d w2v_cc_300d Embeddings Malay (macrolanguage) WordEmbeddingsModel ms 639-2/T: msa639-2/B: may msa Living Macrolanguage
615 mn.embed.w2v_cc_300d w2v_cc_300d Embeddings Mongolian WordEmbeddingsModel mn mon mon Living Macrolanguage
616 ne.embed.w2v_cc_300d w2v_cc_300d Embeddings Nepali (macrolanguage) WordEmbeddingsModel ne nep nep Living Macrolanguage
617 ne.stopwords stopwords_iso Stop Words Removal Nepali (macrolanguage) StopWordsCleaner ne nep nep Living Macrolanguage
618 no.lemma lemma_nynorsk Lemmatization Norwegian LemmatizerModel no nor nor Living Macrolanguage
619 no.pos pos_bokmaal Part of Speech Tagging Norwegian PerceptronModel no nor nor Living Macrolanguage
620 no.pos pos_bokmaal Part of Speech Tagging Norwegian PerceptronModel no nor nor Living Macrolanguage
621 no.pos pos_bokmaal Part of Speech Tagging Norwegian PerceptronModel no nor nor Living Macrolanguage
622 no.embed.w2v_cc_300d w2v_cc_300d Embeddings Norwegian WordEmbeddingsModel no nor nor Living Macrolanguage
623 no.lemma lemma_nynorsk Lemmatization Norwegian LemmatizerModel no nor nor Living Macrolanguage
624 or.embed.w2v_cc_300d w2v_cc_300d Embeddings Oriya (macrolanguage) WordEmbeddingsModel or ori ori Living Macrolanguage
625 ps.embed.w2v_cc_300d w2v_cc_300d Embeddings Pashto, Pushto WordEmbeddingsModel ps pus pus Living Macrolanguage
626 fa.embed.albert albert_embeddings_albert_fa_base_v2 Embeddings Persian AlbertEmbeddings fa 639-2/T: fas639-2/B: per fas Living Macrolanguage
627 fa.embed.distilbert_fa_zwnj_base distilbert_embeddings_distilbert_fa_zwnj_base Embeddings Persian DistilBertEmbeddings fa 639-2/T: fas639-2/B: per fas Living Macrolanguage
628 fa.embed.albert_fa_zwnj_base_v2 albert_embeddings_albert_fa_zwnj_base_v2 Embeddings Persian AlbertEmbeddings fa 639-2/T: fas639-2/B: per fas Living Macrolanguage
629 fa.embed.roberta_fa_zwnj_base roberta_embeddings_roberta_fa_zwnj_base Embeddings Persian RoBertaEmbeddings fa 639-2/T: fas639-2/B: per fas Living Macrolanguage
630 fa.ner.roberta_fa_zwnj_base_ner roberta_ner_roberta_fa_zwnj_base_ner Named Entity Recognition Persian RoBertaForTokenClassification fa 639-2/T: fas639-2/B: per fas Living Macrolanguage
631 fa.pos pos_perdt Part of Speech Tagging Persian PerceptronModel fa 639-2/T: fas639-2/B: per fas Living Macrolanguage
632 fa.stopwords stopwords_iso Stop Words Removal Persian StopWordsCleaner fa 639-2/T: fas639-2/B: per fas Living Macrolanguage
633 qu.embed.w2v_cc_300d w2v_cc_300d Embeddings Quechua WordEmbeddingsModel qu que que Living Macrolanguage
634 sc.embed.w2v_cc_300d w2v_cc_300d Embeddings Sardinian WordEmbeddingsModel sc srd srd Living Macrolanguage
635 sh.embed.w2v_cc_300d w2v_cc_300d Embeddings Serbo-Croatian WordEmbeddingsModel sh nan nan Living Macrolanguage
636 sw.embed.w2v_cc_300d w2v_cc_300d Embeddings Swahili (macrolanguage) WordEmbeddingsModel sw swa swa Living Macrolanguage
637 uz.embed.w2v_cc_300d w2v_cc_300d Embeddings Uzbek WordEmbeddingsModel uz uzb uzb Living Macrolanguage
638 yi.embed.w2v_cc_300d w2v_cc_300d Embeddings Yiddish WordEmbeddingsModel yi yid yid Living Macrolanguage
639 qhe.lemma lemma_hiencs Lemmatization Reserved for local use LemmatizerModel nan qhe qhe nan Local
640 qtd.pos pos_sagt Part of Speech Tagging Reserved for local use PerceptronModel nan qtd qtd nan Local

All Healthcare

Powered by the amazing Spark NLP for Healthcare 3.5.2 and Spark NLP for Healthcare 3.5.1 releases.

Number NLU Reference Spark NLP Reference Task Language Name(s) Annotator Class ISO-639-1 ISO-639-2/639-5 ISO-639-3 Language Type Scope
0 en.med_ner.biomedical_bc2gm ner_biomedical_bc2gm Named Entity Recognition English MedicalNerModel en eng eng Living Individual
1 en.med_ner.biomedical_bc2gm ner_biomedical_bc2gm Named Entity Recognition English MedicalNerModel en eng eng Living Individual
2 en.resolve.rxnorm_action_treatment sbiobertresolve_rxnorm_action_treatment Entity Resolution English SentenceEntityResolverModel en eng eng Living Individual
3 en.classify.token_bert.ner_ade bert_token_classifier_ner_ade Named Entity Recognition English MedicalBertForTokenClassifier en eng eng Living Individual
4 en.classify.token_bert.ner_ade bert_token_classifier_ner_ade Named Entity Recognition English MedicalBertForTokenClassifier en eng eng Living Individual
5 pt.med_ner.deid.subentity ner_deid_subentity De-identification Portuguese MedicalNerModel pt por por Living Individual
6 pt.med_ner.deid.generic ner_deid_generic De-identification Portuguese MedicalNerModel pt por por Living Individual
7 pt.med_ner.deid ner_deid_generic De-identification Portuguese MedicalNerModel pt por por Living Individual

NLU Version 3.4.3

Zero-Shot-Relation-Extraction, DeBERTa for Sequence Classification, 150+ new models, 60+ Languages in John Snow Labs NLU 3.4.3

We are very excited to announce NLU 3.4.3 has been released!

This release features new models for Zero-Shot-Relation-Extraction, DeBERTa for Sequence Classification, Deidentification in French and Italian and Lemmatizers, Parts of Speech Taggers, and Word2Vec Embeddings for over 66 languages, with 20 languages being covered for the first time by NLU, including ancient and exotic languages like Ancient Greek, Old Russian, Old French and much more. Once again we would like to thank our community to make this release possible.

NLU for Healthcare

On the healthcare NLP side, a new ZeroShotRelationExtractionModel is available, which can extract relations between clinical entities in an unsupervised fashion, no training required! Additionally, New French and Italian Deidentification models are available for clinical and healthcare domains. Powerd by the fantastic Spark NLP for helathcare 3.5.0 release

Zero-Shot Relation Extraction

Zero-shot Relation Extraction to extract relations between clinical entities with no training dataset

import nlu

pipe = nlu.load('med_ner.clinical relation.zeroshot_biobert')
# Configure relations to extract
pipe['zero_shot_relation_extraction'].setRelationalCategories({
    "CURE": [" cures ."],
    "IMPROVE": [" improves .", " cures ."],
    "REVEAL": [" reveals ."]})
.setMultiLabel(False)
df = pipe.predict("Paracetamol can alleviate headache or sickness. An MRI test can be used to find cancer.")
df[
    'relation', 'relation_confidence', 'relation_entity1', 'relation_entity1_class', 'relation_entity2', 'relation_entity2_class',]
# Results in following table :
relation relation_confidence relation_entity1 relation_entity1_class relation_entity2 relation_entity2_class
REVEAL 0.976004 An MRI test TEST cancer PROBLEM
IMPROVE 0.988195 Paracetamol TREATMENT sickness PROBLEM
IMPROVE 0.992962 Paracetamol TREATMENT headache PROBLEM

New Healthcare Models overview

Language NLU Reference Spark NLP Reference Task Annotator Class
en en.relation.zeroshot_biobert re_zeroshot_biobert Relation Extraction ZeroShotRelationExtractionModel
fr fr.med_ner.deid_generic ner_deid_generic De-identification MedicalNerModel
fr fr.med_ner.deid_subentity ner_deid_subentity De-identification MedicalNerModel
it it.med_ner.deid_generic ner_deid_generic Named Entity Recognition MedicalNerModel
it it.med_ner.deid_subentity ner_deid_subentity Named Entity Recognition MedicalNerModel

NLU general

On the general NLP side we have new transformer based DeBERTa v3 sequence classifiers models fine-tuned in Urdu, French and English for Sentiment and News classification. Additionally, 100+ Part Of Speech Taggers and Lemmatizers for 66 Languages and for 7 languages new word2vec embeddings, including hi,azb,bo,diq,cy,es,it,
powered by the amazing Spark NLP 3.4.3 release

New Languages covered:

First time languages covered by NLU are : South Azerbaijani, Tibetan, Dimli, Central Kurdish, Southern Altai, Scottish Gaelic,Faroese,Literary Chinese,Ancient Greek, Gothic, Old Russian, Church Slavic, Old French,Uighur,Coptic,Croatian, Belarusian, Serbian

and their respective ISO-639-3 and ISO 630-2 codes are : azb,bo,diq,ckb, lt gd, fo,lzh,grc,got,orv,cu,fro,qtd,ug,cop,hr,be,qhe,sr

New NLP Models Overview

Language NLU Reference Spark NLP Reference Task Annotator Class
en en.classify.sentiment.imdb.deberta deberta_v3_xsmall_sequence_classifier_imdb Text Classification DeBertaForSequenceClassification
en en.classify.sentiment.imdb.deberta.small deberta_v3_small_sequence_classifier_imdb Text Classification DeBertaForSequenceClassification
en en.classify.sentiment.imdb.deberta.base deberta_v3_base_sequence_classifier_imdb Text Classification DeBertaForSequenceClassification
en en.classify.sentiment.imdb.deberta.large deberta_v3_large_sequence_classifier_imdb Text Classification DeBertaForSequenceClassification
en en.classify.news.deberta deberta_v3_xsmall_sequence_classifier_ag_news Text Classification DeBertaForSequenceClassification
en en.classify.news.deberta.small deberta_v3_small_sequence_classifier_ag_news Text Classification DeBertaForSequenceClassification
ur ur.classify.sentiment.imdb mdeberta_v3_base_sequence_classifier_imdb Text Classification DeBertaForSequenceClassification
fr fr.classify.allocine mdeberta_v3_base_sequence_classifier_allocine Text Classification DeBertaForSequenceClassification
ur ur.embed.bert_cased bert_embeddings_bert_base_ur_cased Embeddings BertEmbeddings
fr fr.embed.bert_5lang_cased bert_embeddings_bert_base_5lang_cased Embeddings BertEmbeddings
de de.embed.medbert bert_embeddings_German_MedBERT Embeddings BertEmbeddings
ar ar.embed.arbert bert_embeddings_ARBERT Embeddings BertEmbeddings
bn bn.embed.bangala_bert bert_embeddings_bangla_bert_base Embeddings BertEmbeddings
zh zh.embed.bert_5lang_cased bert_embeddings_bert_base_5lang_cased Embeddings BertEmbeddings
hi hi.embed.bert_hi_cased bert_embeddings_bert_base_hi_cased Embeddings BertEmbeddings
it it.embed.bert_it_cased bert_embeddings_bert_base_it_cased Embeddings BertEmbeddings
ko ko.embed.bert bert_embeddings_bert_base Embeddings BertEmbeddings
tr tr.embed.bert_cased bert_embeddings_bert_base_tr_cased Embeddings BertEmbeddings
vi vi.embed.bert_cased bert_embeddings_bert_base_vi_cased Embeddings BertEmbeddings
hif hif.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel
azb azb.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel
bo bo.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel
diq diq.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel
cy cy.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel
es es.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel
it it.embed.word2vec w2v_cc_300d Embeddings WordEmbeddingsModel
af af.lemma lemma Lemmatization LemmatizerModel
lt lt.lemma lemma_alksnis Lemmatization LemmatizerModel
nl nl.lemma lemma Lemmatization LemmatizerModel
gd gd.lemma lemma_arcosg Lemmatization LemmatizerModel
es es.lemma lemma Lemmatization LemmatizerModel
ca ca.lemma lemma Lemmatization LemmatizerModel
el el.lemma.gdt lemma_gdt Lemmatization LemmatizerModel
en en.lemma.atis lemma_atis Lemmatization LemmatizerModel
tr tr.lemma.boun lemma_boun Lemmatization LemmatizerModel
da da.lemma.ddt lemma_ddt Lemmatization LemmatizerModel
cs cs.lemma.cac lemma_cac Lemmatization LemmatizerModel
en en.lemma.esl lemma_esl Lemmatization LemmatizerModel
bg bg.lemma.btb lemma_btb Lemmatization LemmatizerModel
id id.lemma.csui lemma_csui Lemmatization LemmatizerModel
gl gl.lemma.ctg lemma_ctg Lemmatization LemmatizerModel
cy cy.lemma.ccg lemma_ccg Lemmatization LemmatizerModel
fo fo.lemma.farpahc lemma_farpahc Lemmatization LemmatizerModel
tr tr.lemma.atis lemma_atis Lemmatization LemmatizerModel
ga ga.lemma.idt lemma_idt Lemmatization LemmatizerModel
ja ja.lemma.gsdluw lemma_gsdluw Lemmatization LemmatizerModel
es es.lemma.gsd lemma_gsd Lemmatization LemmatizerModel
en en.lemma.gum lemma_gum Lemmatization LemmatizerModel
zh zh.lemma.gsd lemma_gsd Lemmatization LemmatizerModel
lv lv.lemma.lvtb lemma_lvtb Lemmatization LemmatizerModel
hi hi.lemma.hdtb lemma_hdtb Lemmatization LemmatizerModel
pt pt.lemma.gsd lemma_gsd Lemmatization LemmatizerModel
de de.lemma.gsd lemma_gsd Lemmatization LemmatizerModel
nl nl.lemma.lassysmall lemma_lassysmall Lemmatization LemmatizerModel
lzh lzh.lemma.kyoto lemma_kyoto Lemmatization LemmatizerModel
zh zh.lemma.gsdsimp lemma_gsdsimp Lemmatization LemmatizerModel
he he.lemma.htb lemma_htb Lemmatization LemmatizerModel
fr fr.lemma.gsd lemma_gsd Lemmatization LemmatizerModel
ro ro.lemma.nonstandard lemma_nonstandard Lemmatization LemmatizerModel
ja ja.lemma.gsd lemma_gsd Lemmatization LemmatizerModel
it it.lemma.isdt lemma_isdt Lemmatization LemmatizerModel
de de.lemma.hdt lemma_hdt Lemmatization LemmatizerModel
is is.lemma.modern lemma_modern Lemmatization LemmatizerModel
la la.lemma.ittb lemma_ittb Lemmatization LemmatizerModel
fr fr.lemma.partut lemma_partut Lemmatization LemmatizerModel
pcm pcm.lemma.nsc lemma_nsc Lemmatization LemmatizerModel
pl pl.lemma.pdb lemma_pdb Lemmatization LemmatizerModel
grc grc.lemma.perseus lemma_perseus Lemmatization LemmatizerModel
cs cs.lemma.pdt lemma_pdt Lemmatization LemmatizerModel
fa fa.lemma.perdt lemma_perdt Lemmatization LemmatizerModel
got got.lemma.proiel lemma_proiel Lemmatization LemmatizerModel
fr fr.lemma.rhapsodie lemma_rhapsodie Lemmatization LemmatizerModel
it it.lemma.partut lemma_partut Lemmatization LemmatizerModel
en en.lemma.partut lemma_partut Lemmatization LemmatizerModel
no no.lemma.nynorsklia lemma_nynorsklia Lemmatization LemmatizerModel
orv orv.lemma.rnc lemma_rnc Lemmatization LemmatizerModel
cu cu.lemma.proiel lemma_proiel Lemmatization LemmatizerModel
la la.lemma.perseus lemma_perseus Lemmatization LemmatizerModel
fr fr.lemma.parisstories lemma_parisstories Lemmatization LemmatizerModel
fro fro.lemma.srcmf lemma_srcmf Lemmatization LemmatizerModel
vi vi.lemma.vtb lemma_vtb Lemmatization LemmatizerModel
qtd qtd.lemma.sagt lemma_sagt Lemmatization LemmatizerModel
ro ro.lemma.rrt lemma_rrt Lemmatization LemmatizerModel
hu hu.lemma.szeged lemma_szeged Lemmatization LemmatizerModel
ug ug.lemma.udt lemma_udt Lemmatization LemmatizerModel
wo wo.lemma.wtb lemma_wtb Lemmatization LemmatizerModel
cop cop.lemma.scriptorium lemma_scriptorium Lemmatization LemmatizerModel
ru ru.lemma.syntagrus lemma_syntagrus Lemmatization LemmatizerModel
ru ru.lemma.taiga lemma_taiga Lemmatization LemmatizerModel
fr fr.lemma.sequoia lemma_sequoia Lemmatization LemmatizerModel
la la.lemma.udante lemma_udante Lemmatization LemmatizerModel
ro ro.lemma.simonero lemma_simonero Lemmatization LemmatizerModel
it it.lemma.vit lemma_vit Lemmatization LemmatizerModel
hr hr.lemma.set lemma_set Lemmatization LemmatizerModel
fa fa.lemma.seraji lemma_seraji Lemmatization LemmatizerModel
tr tr.lemma.tourism lemma_tourism Lemmatization LemmatizerModel
ta ta.lemma.ttb lemma_ttb Lemmatization LemmatizerModel
sl sl.lemma.ssj lemma_ssj Lemmatization LemmatizerModel
sv sv.lemma.talbanken lemma_talbanken Lemmatization LemmatizerModel
uk uk.lemma.iu lemma_iu Lemmatization LemmatizerModel
te te.pos pos_mtg Part of Speech Tagging PerceptronModel
te te.pos pos_mtg Part of Speech Tagging PerceptronModel
ta ta.pos pos_ttb Part of Speech Tagging PerceptronModel
ta ta.pos pos_ttb Part of Speech Tagging PerceptronModel
cs cs.pos pos_ud_pdt Part of Speech Tagging PerceptronModel
cs cs.pos pos_ud_pdt Part of Speech Tagging PerceptronModel
bg bg.pos pos_btb Part of Speech Tagging PerceptronModel
bg bg.pos pos_btb Part of Speech Tagging PerceptronModel
af af.pos pos_afribooms Part of Speech Tagging PerceptronModel
af af.pos pos_afribooms Part of Speech Tagging PerceptronModel
af af.pos pos_afribooms Part of Speech Tagging PerceptronModel
es es.pos.gsd pos_gsd Part of Speech Tagging PerceptronModel
en en.pos.ewt pos_ewt Part of Speech Tagging PerceptronModel
gd gd.pos.arcosg pos_arcosg Part of Speech Tagging PerceptronModel
el el.pos.gdt pos_gdt Part of Speech Tagging PerceptronModel
hy hy.pos.armtdp pos_armtdp Part of Speech Tagging PerceptronModel
pt pt.pos.bosque pos_bosque Part of Speech Tagging PerceptronModel
tr tr.pos.framenet pos_framenet Part of Speech Tagging PerceptronModel
cs cs.pos.cltt pos_cltt Part of Speech Tagging PerceptronModel
eu eu.pos.bdt pos_bdt Part of Speech Tagging PerceptronModel
et et.pos.ewt pos_ewt Part of Speech Tagging PerceptronModel
da da.pos.ddt pos_ddt Part of Speech Tagging PerceptronModel
cy cy.pos.ccg pos_ccg Part of Speech Tagging PerceptronModel
lt lt.pos.alksnis pos_alksnis Part of Speech Tagging PerceptronModel
nl nl.pos.alpino pos_alpino Part of Speech Tagging PerceptronModel
fi fi.pos.ftb pos_ftb Part of Speech Tagging PerceptronModel
tr tr.pos.atis pos_atis Part of Speech Tagging PerceptronModel
ca ca.pos.ancora pos_ancora Part of Speech Tagging PerceptronModel
gl gl.pos.ctg pos_ctg Part of Speech Tagging PerceptronModel
de de.pos.gsd pos_gsd Part of Speech Tagging PerceptronModel
fr fr.pos.gsd pos_gsd Part of Speech Tagging PerceptronModel
ja ja.pos.gsdluw pos_gsdluw Part of Speech Tagging PerceptronModel
it it.pos.isdt pos_isdt Part of Speech Tagging PerceptronModel
be be.pos.hse pos_hse Part of Speech Tagging PerceptronModel
nl nl.pos.lassysmall pos_lassysmall Part of Speech Tagging PerceptronModel
sv sv.pos.lines pos_lines Part of Speech Tagging PerceptronModel
uk uk.pos.iu pos_iu Part of Speech Tagging PerceptronModel
fr fr.pos.parisstories pos_parisstories Part of Speech Tagging PerceptronModel
en en.pos.partut pos_partut Part of Speech Tagging PerceptronModel
la la.pos.ittb pos_ittb Part of Speech Tagging PerceptronModel
lzh lzh.pos.kyoto pos_kyoto Part of Speech Tagging PerceptronModel
id id.pos.gsd pos_gsd Part of Speech Tagging PerceptronModel
he he.pos.htb pos_htb Part of Speech Tagging PerceptronModel
tr tr.pos.kenet pos_kenet Part of Speech Tagging PerceptronModel
de de.pos.hdt pos_hdt Part of Speech Tagging PerceptronModel
qhe qhe.pos.hiencs pos_hiencs Part of Speech Tagging PerceptronModel
la la.pos.llct pos_llct Part of Speech Tagging PerceptronModel
en en.pos.lines pos_lines Part of Speech Tagging PerceptronModel
pcm pcm.pos.nsc pos_nsc Part of Speech Tagging PerceptronModel
ko ko.pos.kaist pos_kaist Part of Speech Tagging PerceptronModel
pt pt.pos.gsd pos_gsd Part of Speech Tagging PerceptronModel
hi hi.pos.hdtb pos_hdtb Part of Speech Tagging PerceptronModel
is is.pos.modern pos_modern Part of Speech Tagging PerceptronModel
en en.pos.gum pos_gum Part of Speech Tagging PerceptronModel
fro fro.pos.srcmf pos_srcmf Part of Speech Tagging PerceptronModel
sl sl.pos.ssj pos_ssj Part of Speech Tagging PerceptronModel
ru ru.pos.taiga pos_taiga Part of Speech Tagging PerceptronModel
grc grc.pos.perseus pos_perseus Part of Speech Tagging PerceptronModel
sr sr.pos.set pos_set Part of Speech Tagging PerceptronModel
orv orv.pos.rnc pos_rnc Part of Speech Tagging PerceptronModel
ug ug.pos.udt pos_udt Part of Speech Tagging PerceptronModel
got got.pos.proiel pos_proiel Part of Speech Tagging PerceptronModel
sv sv.pos.talbanken pos_talbanken Part of Speech Tagging PerceptronModel
sv sv.pos.talbanken pos_talbanken Part of Speech Tagging PerceptronModel
pl pl.pos.pdb pos_pdb Part of Speech Tagging PerceptronModel
fa fa.pos.seraji pos_seraji Part of Speech Tagging PerceptronModel
tr tr.pos.penn pos_penn Part of Speech Tagging PerceptronModel
hu hu.pos.szeged pos_szeged Part of Speech Tagging PerceptronModel
sk sk.pos.snk pos_snk Part of Speech Tagging PerceptronModel
sk sk.pos.snk pos_snk Part of Speech Tagging PerceptronModel
ro ro.pos.simonero pos_simonero Part of Speech Tagging PerceptronModel
it it.pos.postwita pos_postwita Part of Speech Tagging PerceptronModel
gl gl.pos.treegal pos_treegal Part of Speech Tagging PerceptronModel
cs cs.pos.pdt pos_pdt Part of Speech Tagging PerceptronModel
ro ro.pos.rrt pos_rrt Part of Speech Tagging PerceptronModel
orv orv.pos.torot pos_torot Part of Speech Tagging PerceptronModel
hr hr.pos.set pos_set Part of Speech Tagging PerceptronModel
la la.pos.proiel pos_proiel Part of Speech Tagging PerceptronModel
fr fr.pos.partut pos_partut Part of Speech Tagging PerceptronModel
it it.pos.vit pos_vit Part of Speech Tagging PerceptronModel

Bugfixes

  • Improved Error Messages and integrated detection and stopping of endless loops which could occur during construction of nlu pipelines

Additional NLU resources

Install NLU in 1 line!

* Install NLU on Google Colab : !wget https://setup.johnsnowlabs.com/nlu/colab.sh  -O - | bash
* Install NLU on Kaggle       : !wget https://setup.johnsnowlabs.com/nlu/kaggle.sh  -O - | bash
* Install NLU via Pip         : ! pip install nlu pyspark streamlit==0.80.0`

NLU Version 3.4.2

Multilingual DeBERTa Transformer Embeddings for 100+ Languages, Spanish Deidentification and NER for Randomized Clinical Trials - John Snow Labs NLU 3.4.2

We are very excited NLU 3.4.2 has been released. On the open source side we have 5 new DeBERTa Transformer models for English and Multi-Lingual for 100+ languages. DeBERTa improves over BERT and RoBERTa by introducing two novel techniques.

For the healthcare side we have new NER models for randomized clinical trials (RCT) which can detect entities of type BACKGROUND, CONCLUSIONS, METHODS, OBJECTIVE, RESULTS from clinical text. Additionally, new Spanish Deidentification NER models for entities like STATE, PATIENT, DEVICE, COUNTRY, ZIP, PHONE, HOSPITAL and many more.

New Open Source Models

Integrates models from Spark NLP 3.4.2 release

Language NLU Reference Spark NLP Reference Task Annotator Class
en en.embed.deberta_v3_xsmall deberta_v3_xsmall Embeddings DeBertaEmbeddings
en en.embed.deberta_v3_small deberta_v3_small Embeddings DeBertaEmbeddings
en en.embed.deberta_v3_base deberta_v3_base Embeddings DeBertaEmbeddings
en en.embed.deberta_v3_large deberta_v3_large Embeddings DeBertaEmbeddings
xx xx.embed.mdeberta_v3_base mdeberta_v3_base Embeddings DeBertaEmbeddings

New Healthcare Models

Integrates models from Spark NLP For Healthcare 3.4.2 release

Language NLU Reference Spark NLP Reference Task Annotator Class
en en.med_ner.clinical_trials bert_sequence_classifier_rct_biobert Text Classification MedicalBertForSequenceClassification
es es.med_ner.deid.generic.roberta ner_deid_generic_roberta_augmented De-identification MedicalNerModel
es es.med_ner.deid.subentity.roberta ner_deid_subentity_roberta_augmented De-identification MedicalNerModel
en en.med_ner.deid.generic_augmented ner_deid_generic_augmented [‘Named Entity Recognition’, ‘De-identification’] MedicalNerModel
en en.med_ner.deid.subentity_augmented ner_deid_subentity_augmented [‘Named Entity Recognition’, ‘De-identification’] MedicalNerModel

Additional NLU resources

Install NLU in 1 line!

* Install NLU on Google Colab : !wget https://setup.johnsnowlabs.com/nlu/colab.sh  -O - | bash
* Install NLU on Kaggle       : !wget https://setup.johnsnowlabs.com/nlu/kaggle.sh  -O - | bash
* Install NLU via Pip         : ! pip install nlu pyspark streamlit==0.80.0`

NLU Version 3.4.1

22 New models for 23 languages including various African and Indian languages, Medical Spanish models and more in NLU 3.4.1

We are very excited to announce the release of NLU 3.4.1 which features 22 new models for 23 languages where the The open-source side covers new Embeddings for Vietnamese and English Clinical domains and Multilingual Embeddings for 12 Indian and 9 African Languages. Additionally, there are new Sequence classifiers for Multilingual NER for 9 African languages, German Sentiment Classifiers and English Emotion and Typo Classifiers. The healthcare side covers Medical Spanish models, Classifiers for Drugs, Gender, the Pico Framework, and Relation Extractors for Adverse Drug events and Temporality. Finally, Spark 3.2.X is now supported and bugs related to Databricks environments have been fixed.

General NLU Improvements

  • Support for Spark 3.2.x

New Open Source Models

Based on the amazing 3.4.1 Spark NLP Release integrates new Multilingual embeddings for 12 Major Indian languages, embeddings for Vietnamese, French, and English Clinical domains. Additionally new Multilingual NER model for 9 African languages, English 6 Class Emotion classifier and Typo detectors.

New Embeddings

  • Multilingual ALBERT - IndicBert model pretrained exclusively on 12 major Indian languages with size smaller and performance on par or better than competing models. Languages covered are Assamese, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu. Available with xx.embed.albert.indic
  • Fine tuned Vietnamese DistilBERT Base cased embeddings. Available with vi.embed.distilbert.cased
  • Clinical Longformer Embeddings which consistently out-performs ClinicalBERT for various downstream tasks and on datasets. Available with en.embed.longformer.clinical
  • Fine tuned Static French Word2Vec Embeddings in 3 sizes, 200d, 300d and 100d. Available with fr.embed.word2vec_wiki_1000, fr.embed.word2vec_wac_200 and fr.embed.w2v_cc_300d

New Transformer based Token and Sequence Classifiers

Language NLU Reference Spark NLP Reference Task Annotator Class
xx xx.embed.albert.indic albert_indic Embeddings AlbertEmbeddings
xx xx.ner.masakhaner.distilbert xlm_roberta_large_token_classifier_masakhaner Named Entity Recognition DistilBertForTokenClassification
en en.embed.longformer.clinical clinical_longformer Embeddings LongformerEmbeddings
en en.classify.emotion.bert bert_sequence_classifier_emotion Text Classification BertForSequenceClassification
de de.classify.news_sentiment.bert bert_sequence_classifier_news_sentiment Sentiment Analysis BertForSequenceClassification
en en.classify.typos.distilbert distilbert_token_classifier_typo_detector Named Entity Recognition DistilBertForTokenClassification
fr fr.embed.word2vec_wiki_1000 word2vec_wiki_1000 Embeddings WordEmbeddingsModel
fr fr.embed.word2vec_wac_200 word2vec_wac_200 Embeddings WordEmbeddingsModel
fr fr.embed.w2v_cc_300d w2v_cc_300d Embeddings WordEmbeddingsModel
vi vi.embed.distilbert.cased distilbert_base_cased Embeddings DistilBertEmbeddings

New Healthcare Models

Integrated from the amazing 3.4.1 Spark NLP For Healthcare Release. which makes 2 new Annotator Classes available, MedicalBertForSequenceClassification and MedicalDistilBertForSequenceClassification, various medical Spanish models, RxNorm Resolvers, Transformer based sequence classifiers for Drugs, Gender and the PICO framework, and Relation extractors for Temporality and Causality of Drugs and Adverse Events.

New Medical Spanish Models

New Resolvers

New Transformer based Sequence Classifiers

New Relation Extractors

Language NLU Reference Spark NLP Reference Task Annotator Class
es es.embed.sciwiki_300d embeddings_sciwiki_300d Embeddings WordEmbeddingsModel
es es.med_ner.deid.generic ner_deid_generic De-identification MedicalNerModel
es es.med_ner.deid.subentity ner_deid_subentity De-identification MedicalNerModel
en en.med_ner.supplement_clinical ner_supplement_clinical Named Entity Recognition MedicalNerModel
en en.resolve.rxnorm.augmented_re sbiobertresolve_rxnorm_augmented_re Entity Resolution SentenceEntityResolverModel
en en.classify.ade.seq_biobert bert_sequence_classifier_ade Text Classification MedicalBertForSequenceClassification
en en.classify.gender.seq_biobert bert_sequence_classifier_gender_biobert Text Classification MedicalBertForSequenceClassification
en en.classify.pico.seq_biobert bert_sequence_classifier_pico_biobert Text Classification MedicalBertForSequenceClassification
en en.classify.ade.seq_distilbert distilbert_sequence_classifier_ade Text Classification MedicalDistilBertForSequenceClassification
en en.relation.temporal_events_clinical re_temporal_events_clinical Relation Extraction RelationExtractionModel
en en.relation.adverse_drug_events.clinical re_ade_clinical Relation Extraction RelationExtractionModel
en en.relation.adverse_drug_events.clinical.biobert redl_ade_biobert Relation Extraction RelationExtractionDLModel

Bugfixes

  • Fixed bug that caused non-default output level of components to be sentence
  • Fixed a bug that caused nlu references pointing to pretrained pipelines in spark nlp to crash in Databricks environments

Additional NLU resources

Install NLU in 1 line!

* Install NLU on Google Colab : !wget https://setup.johnsnowlabs.com/nlu/colab.sh  -O - | bash
* Install NLU on Kaggle       : !wget https://setup.johnsnowlabs.com/nlu/kaggle.sh  -O - | bash
* Install NLU via Pip         : ! pip install nlu pyspark streamlit==0.80.0`

NLU Version 3.4.0

1 line to OCR for images, PDFS and DOCX, Text Generation with GPT2 and new T5 models, Sequence Classification with XlmRoBerta, RoBerta, Xlnet, Longformer and Albert, Transformer based medical NER with MedicalBertForTokenClassifier, 80 new models, 20+ new languages including various African and Scandinavian and much more in John Snow Labs NLU 3.4.0 !

We are incredibly excited to announce John Snow Labs NLU 3.4.0 has been released! This release features 11 new annotator classes and 80 new models, including 3 OCR Transformers which enable you to extract text from various file types, support for GPT2 and new pretrained T5 models for Text Generation and dozens more of new transformer based models for Token and Sequence Classification. This includes 8 new Sequence classifier models which can be pretrained in Huggingface and imported into Spark NLP and NLU. Finally, the NLU tutorial page of the 140+ notebooks has been updated

New NLU OCR Features

3 new OCR based spells are supported, which enable extracting text from files of type JPEG, PNG, BMP, WBMP, GIF, JPG, TIFF, DOCX, PDF in just 1 line of code. You need a Spark OCR license for using these, which is available for free here and refer to the new OCR tutorial notebook
Open In Colab Find more details on the NLU OCR documentation page

New NLU Healthcare Features

The healthcare side features a new MedicalBertForTokenClassifier annotator which is a Bert based model for token classification problems like Named Entity Recognition,
Parts of Speech and much more. Overall there are 28 new models which include German De-Identification models, English NER models for extracting Drug Development Trials,
Clinical Abbreviations and Acronyms, NER models for chemical compounds/drugs and genes/proteins, updated MedicalBertForTokenClassifier NER models for the medical domains Adverse drug Events,
Anatomy, Chemicals, Genes,Proteins, Cellular/Molecular Biology, Drugs, Bacteria, De-Identification and general Medical and Clinical Named Entities.
For Entity Relation Extraction between entity pairs new models for interaction between Drugs and Proteins.
For Entity Resolution new models for resolving Clinical Abbreviations and Acronyms to their full length names and also a model for resolving Drug Substance Entities to the categories
Clinical Drug, Pharmacologic Substance, Antibiotic, Hazardous or Poisonous Substance and new resolvers for LOINC and SNOMED terminologies.

New NLU Open source Features

On the open source side we have new support for Open Ai’s GPT2 for various text sequence to sequence problems and additionally the following new Transformer models are supported : RoBertaForSequenceClassification, XlmRoBertaForSequenceClassification, LongformerForSequenceClassification, AlbertForSequenceClassification, XlnetForSequenceClassification, Word2Vec with various pre-trained weights for various problems!

New GPT2 models for generating text conditioned on some input,
New T5 style transfer models for active to passive, formal to informal, informal to formal, passive to active sequence to sequence generation.
Additionally, a new T5 model for generating SQL code from natural language input is provided.

On top of this dozens new Transformer based Sequence Classifiers and Token Classifiers have been released, this is includes for Token Classifier the following models :
Multi-Lingual general NER models for 10 African Languages (Amharic, Hausa, Igbo, Kinyarwanda, Luganda, Nigerian, Pidgin, Swahilu, Wolof, and Yorùbá),
10 high resourced languages (10 high resourced languages (Arabic, German, English, Spanish, French, Italian, Latvian, Dutch, Portuguese and Chinese),
6 Scandinavian languages (Danish, Norwegian-Bokmål, Norwegian-Nynorsk, Swedish, Icelandic, Faroese) ,
Uni-Lingual NER models for general entites in the language Chinese, Hindi, Islandic, Indonesian
and finally English NER models for extracting entities related to Stocks Ticker Symbols, Restaurants, Time.

For Sequence Classification new models for classifying Toxicity in Russian text and English models for Movie Reviews, News Categorization, Sentimental Tone and General Sentiment

New NLU OCR Models

The following Transformers have been integrated from Spark OCR

NLU Spell Transformer Class
nlu.load(img2text) ImageToText
nlu.load(pdf2text) PdfToText
nlu.load(doc2text) DocToText

New Open Source Models

Integration for the 49 new models from the colossal Spark NLP 3.4.0 release

Language NLU Reference Spark NLP Reference Task Annotator Class
en en.gpt2.distilled gpt2_distilled Text Generation GPT2Transformer
en en.gpt2 gpt2 Text Generation GPT2Transformer
en en.gpt2.medium gpt2_medium Text Generation GPT2Transformer
en en.gpt2.large gpt_large Text Generation GPT2Transformer
en en.t5.active_to_passive_styletransfer t5_active_to_passive_styletransfer Text Generation T5Transformer
en en.t5.formal_to_informal_styletransfer t5_formal_to_informal_styletransfer Text Generation T5Transformer
en en.t5.grammar_error_corrector t5_grammar_error_corrector Text Generation T5Transformer
en en.t5.informal_to_formal_styletransfer t5_informal_to_formal_styletransfer Text Generation T5Transformer
en en.t5.passive_to_active_styletransfer t5_passive_to_active_styletransfer Text Generation T5Transformer
en en.t5.wikiSQL t5_small_wikiSQL Text Generation T5Transformer
xx xx.ner.masakhaner xlm_roberta_large_token_classifier_masakhaner Named Entity Recognition XlmRoBertaForTokenClassification
xx xx.ner.high_resourced_lang xlm_roberta_large_token_classifier_hrl Named Entity Recognition XlmRoBertaForTokenClassification
xx xx.ner.scandinavian bert_token_classifier_scandi_ner Named Entity Recognition BertForTokenClassification
en en.embed.electra.medical electra_medal_acronym Embeddings BertEmbeddings
en en.ner.restaurant nerdl_restaurant_100d Named Entity Recognition NerDLModel
en en.embed.word2vec.gigaword_wiki word2vec_gigaword_wiki_300 Embeddings Word2VecModel
en en.embed.word2vec.gigaword word2vec_gigaword_300 Embeddings Word2VecModel
en en.classify.xlm_roberta.imdb xlm_roberta_base_sequence_classifier_imdb Text Classification XlmRoBertaForSequenceClassification
en en.classify.xlm_roberta.ag_news xlm_roberta_base_sequence_classifier_ag_news Text Classification XlmRoBertaForSequenceClassification
en en.classify.roberta.imdb roberta_base_sequence_classifier_imdb Text Classification RoBertaForSequenceClassification
en en.classify.roberta.ag_news roberta_base_sequence_classifier_ag_news Text Classification RoBertaForSequenceClassification
en en.classify.albert.ag_news albert_base_sequence_classifier_ag_news Text Classification AlbertForSequenceClassification
en en.classify.albert.imdb albert_base_sequence_classifier_imdb Text Classification AlbertForSequenceClassification
en en.classify.ag_news.longformer longformer_base_sequence_classifier_ag_news Text Classification LongformerForSequenceClassification
en en.classify.imdb.xlnet xlnet_base_sequence_classifier_imdb Text Classification XlnetForSequenceClassification
en en.classify.finance_sentiment bert_sequence_classifier_finbert_tone Sentiment Analysis BertForSequenceClassification
en en.classify.imdb.longformer longformer_base_sequence_classifier_imdb Text Classification LongformerForSequenceClassification
en en.classify.ag_news.longformer longformer_base_sequence_classifier_ag_news Text Classification LongformerForSequenceClassification
en en.ner.time roberta_token_classifier_timex_semeval Named Entity Recognition RoBertaForTokenClassification
en en.ner.stocks_ticker roberta_token_classifier_ticker Named Entity Recognition RoBertaForTokenClassification
ru ru.classify.toxic bert_sequence_classifier_toxicity Text Classification BertForSequenceClassification
it it.classify.sentiment bert_sequence_classifier_sentiment Sentiment Analysis BertForSequenceClassification
es es.ner wikiner_6B_100 Named Entity Recognition NerDLModel
is is.ner roberta_token_classifier_icelandic_ner Named Entity Recognition RoBertaForTokenClassification
id id.pos roberta_token_classifier_pos_tagger Part of Speech Tagging RoBertaForTokenClassification
tr tr.ner turkish_ner_840B_300 Named Entity Recognition NerDLModel
id id.ner xlm_roberta_large_token_classification_ner Named Entity Recognition XlmRoBertaForTokenClassification
de de.ner xlm_roberta_large_token_classifier_conll03 Named Entity Recognition XlmRoBertaForTokenClassification
hi hi.ner bert_token_classifier_hi_en_ner Named Entity Recognition BertForTokenClassification
nl nl.ner wikiner_6B_100 Named Entity Recognition NerDLModel
zh zh.ner bert_token_classifier_chinese_ner Named Entity Recognition BertForTokenClassification
fr fr.classify.xlm_roberta.allocine xlm_roberta_base_sequence_classifier_allocine Text Classification XlmRoBertaForSequenceClassification
ur ur.classify.fakenews classifierdl_urduvec_fakenews Text Classification ClassifierDLModel
ur ur.classify.news classifierdl_bert_news Text Classification ClassifierDLModel
fi fi.embed_sentence.bert.uncased bert_base_finnish_uncased Embeddings BertSentenceEmbeddings
fi fi.embed_sentence.bert bert_base_finnish_uncased Embeddings BertSentenceEmbeddings
fi fi.embed_sentence.bert.cased bert_base_finnish_cased Embeddings BertSentenceEmbeddings
te te.embed.distilbert distilbert_uncased Embeddings DistilBertEmbeddings
sw sw.embed.xlm_roberta xlm_roberta_base_finetuned_swahili Embeddings XlmRoBertaEmbeddings

New Healthcare Models

Integration for the 28 new models from the amazing Spark NLP for healthcare 3.4.0 release

Language NLU Reference Spark NLP Reference Task Annotator Class
en en.med_ner.chemprot.bert bert_token_classifier_ner_chemprot Named Entity Recognition MedicalBertForTokenClassifier
en en.med_ner.chemprot.bert bert_token_classifier_ner_chemprot Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_bacteria bert_token_classifier_ner_bacteria Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_bacteria bert_token_classifier_ner_bacteria Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_anatomy bert_token_classifier_ner_anatomy Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_anatomy bert_token_classifier_ner_anatomy Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_drugs bert_token_classifier_ner_drugs Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_drugs bert_token_classifier_ner_drugs Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_jsl_slim bert_token_classifier_ner_jsl_slim Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_jsl_slim bert_token_classifier_ner_jsl_slim Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_ade bert_token_classifier_ner_ade Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_ade bert_token_classifier_ner_ade Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_deid bert_token_classifier_ner_deid Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_deid bert_token_classifier_ner_deid Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_clinical bert_token_classifier_ner_clinical Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_clinical bert_token_classifier_ner_clinical Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_jsl bert_token_classifier_ner_jsl Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_jsl bert_token_classifier_ner_jsl Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_jsl bert_token_classifier_ner_jsl Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_chemical bert_token_classifier_ner_chemicals Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.ner_chemical bert_token_classifier_ner_chemicals Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.bionlp bert_token_classifier_ner_bionlp Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.bionlp bert_token_classifier_ner_bionlp Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.cellular bert_token_classifier_ner_cellular Named Entity Recognition MedicalBertForTokenClassifier
en en.classify.token_bert.cellular bert_token_classifier_ner_cellular Named Entity Recognition MedicalBertForTokenClassifier
en en.med_ner.abbreviation_clinical ner_abbreviation_clinical Named Entity Recognition MedicalNerModel
en en.med_ner.drugprot_clinical ner_drugprot_clinical Named Entity Recognition MedicalNerModel
en en.ner.drug_development_trials bert_token_classifier_drug_development_trials Named Entity Recognition BertForTokenClassification
en en.med_ner.chemprot ner_chemprot_biobert Named Entity Recognition MedicalNerModel
en en.relation.drugprot redl_drugprot_biobert Relation Extraction RelationExtractionDLModel
en en.relation.drugprot.clinical re_drugprot_clinical Relation Extraction RelationExtractionModel
en en.resolve.clinical_abbreviation_acronym sbiobertresolve_clinical_abbreviation_acronym Entity Resolution SentenceEntityResolverModel
en en.resolve.clinical_abbreviation_acronym sbiobertresolve_clinical_abbreviation_acronym Entity Resolution SentenceEntityResolverModel
en en.resolve.umls_drug_substance sbiobertresolve_umls_drug_substance Entity Resolution SentenceEntityResolverModel
en en.resolve.loinc_cased sbiobertresolve_loinc_cased Entity Resolution SentenceEntityResolverModel
en en.resolve.loinc_uncased sbluebertresolve_loinc_uncased Entity Resolution SentenceEntityResolverModel
en en.embed_sentence.biobert.rxnorm sbiobert_jsl_rxnorm_cased Entity Resolution BertSentenceEmbeddings
en en.embed_sentence.bert_uncased.rxnorm sbert_jsl_medium_rxnorm_uncased Embeddings BertSentenceEmbeddings
en en.embed_sentence.bert_uncased.rxnorm sbert_jsl_medium_rxnorm_uncased Embeddings BertSentenceEmbeddings
en en.resolve.snomed_drug sbiobertresolve_snomed_drug Entity Resolution SentenceEntityResolverModel
de de.med_ner.deid_subentity ner_deid_subentity Named Entity Recognition MedicalNerModel
de de.med_ner.deid_generic ner_deid_generic Named Entity Recognition MedicalNerModel
de de.embed.w2v w2v_cc_300d Embeddings WordEmbeddingsModel

Additional NLU resources

Install NLU in 1 line!

* Install NLU on Google Colab : !wget https://setup.johnsnowlabs.com/nlu/colab.sh  -O - | bash
* Install NLU on Kaggle       : !wget https://setup.johnsnowlabs.com/nlu/kaggle.sh  -O - | bash
* Install NLU via Pip         : ! pip install nlu pyspark streamlit==0.80.0`

NLU Version 3.3.1

48 new Transformer based models in 9 new languages, including NER for Finance, Industry, Politcal Policies, COVID and Chemical Trials, various clinical and medical domains in Spanish and English and much more in NLU 3.3.1

We are incredibly excited to announce NLU 3.3.1 has been released with 48 new models in 9 languages!

It comes with 2 new types of state-of-the-art models,distilBERT and BERT for sequence classification with various pre-trained weights, state-of-the-art bert based classifiers for problems in the domains of Finance, Sentiment Classification, Industry, News, and much more.

On the healthcare side, NLU features 22 new models in for English and Spanish with with entity Resolver Models for LOINC, MeSH, NDC and SNOMED and UMLS Diseases, NER models for Biomarkers, NIHSS-Guidelines, COVID Trials , Chemical Trials, Bert based Token Classifier models for biological, genetical,cancer, cellular terms, Bert for Sequence Classification models for clinical question vs statement classification and finally Spanish Clinical NER and Resolver Models

Once again, we would like to thank our community for making another amazing release possible!

New Open Source Models and Features

Integrates the amazing Spark NLP 3.3.3 and 3.3.2 releases, featuring:

  • New state-of-the-art fine-tuned BERT models for Sequence Classification in English, French, German, Spanish, Japanese, Turkish, Russian, and multilingual languages.
  • DistilBertForSequenceClassification models in English, French and Urdu
  • Word2Vec models.
  • classify.distilbert_sequence.banking77 : Banking NER model trained on BANKING77 dataset, which provides a very fine-grained set of intents in a banking domain. It comprises 13,083 customer service queries labeled with 77 intents. It focuses on fine-grained single-domain intent detection. Can extract entities like activate_my_card, age_limit, apple_pay_or_google_pay, atm_support, automatic_top_up, balance_not_updated_after_bank_transfer, balance_not_updated_after_cheque_or_cash_deposit, beneficiary_not_allowed, cancel_transfer, card_about_to_expire, card_acceptance, card_arrival, card_delivery_estimate, card_linking, card_not_working, card_payment_fee_charged, card_payment_not_recognised, card_payment_wrong_exchange_rate, card_swallowed, cash_withdrawal_charge, cash_withdrawal_not_recognised, change_pin, compromised_card, contactless_not_working, country_support, declined_card_payment, declined_cash_withdrawal, declined_transfer, direct_debit_payment_not_recognised, disposable_card_limits, edit_personal_details, exchange_charge, exchange_rate, exchange_via_app, extra_charge_on_statement, failed_transfer, fiat_currency_support, get_disposable_virtual_card, get_physical_card, getting_spare_card, getting_virtual_card, lost_or_stolen_card, lost_or_stolen_phone, order_physical_card, passcode_forgotten, pending_card_payment, pending_cash_withdrawal, pending_top_up, pending_transfer, pin_blocked, receiving_money,
  • classify.distilbert_sequence.industry : Industry NER model which can extract entities like Advertising, Aerospace & Defense, Apparel Retail, Apparel, Accessories & Luxury Goods, Application Software, Asset Management & Custody Banks, Auto Parts & Equipment, Biotechnology, Building Products, Casinos & Gaming, Commodity Chemicals, Communications Equipment, Construction & Engineering, Construction Machinery & Heavy Trucks, Consumer Finance, Data Processing & Outsourced Services, Diversified Metals & Mining, Diversified Support Services, Electric Utilities, Electrical Components & Equipment, Electronic Equipment & Instruments, Environmental & Facilities Services, Gold, Health Care Equipment, Health Care Facilities, Health Care Services.
  • xx.classify.bert_sequence.sentiment : Multi-Lingual Sentiment Classifier This a bert-base-multilingual-uncased model finetuned for sentiment analysis on product reviews in six languages: English, Dutch, German, French, Spanish and Italian. It predicts the sentiment of the review as a number of stars (between 1 and 5). This model is intended for direct use as a sentiment analysis model for product reviews in any of the six languages above, or for further finetuning on related sentiment analysis tasks.
  • distilbert_sequence.policy : Policy Classifier This model was trained on 129.669 manually annotated sentences to classify text into one of seven political categories: ‘Economy’, ‘External Relations’, ‘Fabric of Society’, ‘Freedom and Democracy’, ‘Political System’, ‘Welfare and Quality of Life’ or ‘Social Groups’.
  • classify.bert_sequence.dehatebert_mono : Hate Speech Classifier This model was trained on 129.669 manually annotated sentences to classify text into one of seven political categories: ‘Economy’, ‘External Relations’, ‘Fabric of Society’, ‘Freedom and Democracy’, ‘Political System’, ‘Welfare and Quality of Life’ or ‘Social Groups’.

Complete List of Open Source Models :

| Language | NLU Reference | Spark NLP Reference | Task | |:———–|:——————————————————————————————————————————————————-|:————————————————————————————————————————————————————-|:——————–| | en | en.classify.bert_sequence.imdb_large | bert_large_sequence_classifier_imdb | Text Classification | | en | en.classify.bert_sequence.imdb | bert_base_sequence_classifier_imdb | Text Classification | | en | en.classify.bert_sequence.ag_news | bert_base_sequence_classifier_ag_news | Text Classification | | en | en.classify.bert_sequence.dbpedia_14 | bert_base_sequence_classifier_dbpedia_14 | Text Classification | | en | en.classify.bert_sequence.finbert | bert_sequence_classifier_finbert | Text Classification | | en | en.classify.bert_sequence.dehatebert_mono | bert_sequence_classifier_dehatebert_mono | Text Classification | | tr | tr.classify.bert_sequence.sentiment | bert_sequence_classifier_turkish_sentiment | Text Classification | | de | de.classify.bert_sequence.sentiment | bert_sequence_classifier_sentiment | Text Classification | | ru | ru.classify.bert_sequence.sentiment | bert_sequence_classifier_rubert_sentiment | Text Classification | | ja | ja.classify.bert_sequence.sentiment | bert_sequence_classifier_japanese_sentiment | Text Classification | | es | es.classify.bert_sequence.sentiment | bert_sequence_classifier_beto_sentiment_analysis | Text Classification | | es | es.classify.bert_sequence.emotion | bert_sequence_classifier_beto_emotion_analysis | Text Classification | | xx | xx.classify.bert_sequence.sentiment | bert_sequence_classifier_multilingual_sentiment | Text Classification | | en | en.classify.distilbert_sequence.sst2 | distilbert_sequence_classifier_sst2 | Text Classification | | en | en.classify.distilbert_sequence.policy | distilbert_sequence_classifier_policy | Text Classification | | en | en.classify.distilbert_sequence.industry | distilbert_sequence_classifier_industry | Text Classification | | en | en.classify.distilbert_sequence.emotion | distilbert_sequence_classifier_emotion | Text Classification | | en | en.classify.distilbert_sequence.banking77 | distilbert_sequence_classifier_banking77 | Text Classification | | en | en.classify.distilbert_sequence.imdb | distilbert_base_sequence_classifier_imdb | Text Classification | | en | en.classify.distilbert_sequence.amazon_polarity | distilbert_base_sequence_classifier_amazon_polarity | Text Classification | | en | en.classify.distilbert_sequence.ag_news | distilbert_base_sequence_classifier_ag_news | Text Classification | | fr | fr.classify.distilbert_sequence.allocine | distilbert_multilingual_sequence_classifier_allocine | Text Classification | | ur | ur.classify.distilbert_sequence.imdb | distilbert_base_sequence_classifier_imdb | Text Classification | | en | en.embed_sentence.doc2vec | doc2vec_gigaword_300 | Embeddings | | en | en.embed_sentence.doc2vec.gigaword_300 | doc2vec_gigaword_300 | Embeddings | | en | en.embed_sentence.doc2vec.gigaword_wiki_300 | doc2vec_gigaword_wiki_300 | Embeddings |

New Healthcare models and Features

Integrates the incredible Spark NLP for Healthcare releases 3.3.4, 3.3.2 and 3.3.1, featuring:

  • New Clinical NER Models for protected health information(PHI),
    • ner_biomarker for extracting extract biomarkers, therapies, oncological, and other general concepts
      • Oncogenes, Tumor_Finding, UnspecificTherapy, Ethnicity, Age, ResponseToTreatment, Biomarker, HormonalTherapy, Staging, Drug, CancerDx, Radiotherapy, CancerSurgery, TargetedTherapy, PerformanceStatus, CancerModifier, Radiological_Test_Result, Biomarker_Measurement, Metastasis, Radiological_Test, Chemotherapy, Test, Dosage, Test_Result, Immunotherapy, Date, Gender, Prognostic_Biomarkers, Duration, Predictive_Biomarkers
  • ner_nihss : NER model that can identify entities according to NIHSS guidelines for clinical stroke assessment to evaluate neurological status in acute stroke patients
    • 11_ExtinctionInattention, 6b_RightLeg, 1c_LOCCommands, 10_Dysarthria, NIHSS, 5_Motor, 8_Sensory, 4_FacialPalsy, 6_Motor, 2_BestGaze, Measurement, 6a_LeftLeg, 5b_RightArm, 5a_LeftArm, 1b_LOCQuestions, 3_Visual, 9_BestLanguage, 7_LimbAtaxia, 1a_LOC .
  • redl_nihss_biobert : relation extraction model that can relate scale items and their measurements according to NIHSS guidelines.
  • es.med_ner.roberta_ner_diag_proc : New Spanish Clinical NER Models for extracting the entities DIAGNOSTICO, PROCEDIMIENTO
  • es.resolve.snomed: New Spanish SNOMED Entity Resolvers
  • bert_sequence_classifier_question_statement_clinical:New Clinical Question vs Statement for BertForSequenceClassification model
  • med_ner.covid_trials : This model is trained to extract covid-specific medical entities in clinical trials. It supports the following entities ranging from virus type to trial design: Stage, Severity, Virus, Trial_Design, Trial_Phase, N_Patients, Institution, Statistical_Indicator, Section_Header, Cell_Type, Cellular_component, Viral_components, Physiological_reaction, Biological_molecules, Admission_Discharge, Age, BMI, Cerebrovascular_Disease, Date, Death_Entity, Diabetes, Disease_Syndrome_Disorder, Dosage, Drug_Ingredient, Employment, Frequency, Gender, Heart_Disease, Hypertension, Obesity, Pulse, Race_Ethnicity, Respiration, Route, Smoking, Time, Total_Cholesterol, Treatment, VS_Finding, Vaccine .
  • med_ner.chemd : This model extract the names of chemical compounds and drugs in medical texts. The entities that can be detected are as follows : SYSTEMATIC, IDENTIFIERS, FORMULA, TRIVIAL, ABBREVIATION, FAMILY, MULTIPLE . For reference click here . https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4331685/
  • bert_token_classifier_ner_bionlp : This model is BERT-based version of ner_bionlp model and can detect biological and genetics terms in cancer-related texts. (Amino_acid, Anatomical_system, Cancer, Cell, Cellular_component, Developing_anatomical_Structure, Gene_or_gene_product, Immaterial_anatomical_entity, Multi-tissue_structure, Organ, Organism, Organism_subdivision, Simple_chemical, Tissue
  • bert_token_classifier_ner_cellular : This model is BERT-based version of ner_cellular model and can detect molecular biology-related terms (DNA, Cell_type, Cell_line, RNA, Protein) in medical texts.
  • We have updated med_ner.jsl.enriched model by enriching the training data using clinical trials data to make it more robust. This model is capable of predicting up to 87 different entities and is based on ner_jsl model. Here are the entities this model can detect; Social_History_Header, Oncology_Therapy, Blood_Pressure, Respiration, Performance_Status, Family_History_Header, Dosage, Clinical_Dept, Diet, Procedure, HDL, Weight, Admission_Discharge, LDL, Kidney_Disease, Oncological, Route, Imaging_Technique, Puerperium, Overweight, Temperature, Diabetes, Vaccine, Age, Test_Result, Employment, Time, Obesity, EKG_Findings, Pregnancy, Communicable_Disease, BMI, Strength, Tumor_Finding, Section_Header, RelativeDate, ImagingFindings, Death_Entity, Date, Cerebrovascular_Disease, Treatment, Labour_Delivery, Pregnancy_Delivery_Puerperium, Direction, Internal_organ_or_component, Psychological_Condition, Form, Medical_Device, Test, Symptom, Disease_Syndrome_Disorder, Staging, Birth_Entity, Hyperlipidemia, O2_Saturation, Frequency, External_body_part_or_region, Drug_Ingredient, Vital_Signs_Header, Substance_Quantity, Race_Ethnicity, VS_Finding, Injury_or_Poisoning, Medical_History_Header, Alcohol, Triglycerides, Total_Cholesterol, Sexually_Active_or_Sexual_Orientation, Female_Reproductive_Status, Relationship_Status, Drug_BrandName, RelativeTime, Duration, Hypertension, Metastasis, Gender, Oxygen_Therapy, Pulse, Heart_Disease, Modifier, Allergen, Smoking, Substance, Cancer_Modifier, Fetus_NewBorn, Height
  • classify.bert_sequence.question_statement_clinical : This model classifies sentences into one of these two classes: question (interrogative sentence) or statement (declarative sentence) and trained with BertForSequenceClassification. This model is at first trained on SQuAD and SPAADIA dataset and then fine tuned on the clinical visit documents and MIMIC-III dataset annotated in-house. Using this model, you can find the question statements and exclude & utilize in the downstream tasks such as NER and relation extraction models.
  • classify.token_bert.ner_chemical : This model is BERT-based version of ner_chemicals model and can detect chemical compounds (CHEM) in the medical texts.
  • resolve.umls_disease_syndrome : This model is trained on the Disease or Syndrome category using sbiobert_base_cased_mli embeddings.

Complete List of Healthcare Models :

Language NLU Reference Spark NLP Reference Task
en en.med_ner.deid_subentity_augmented_i2b2 ner_deid_subentity_augmented_i2b2 Named Entity Recognition
en en.med_ner.biomarker ner_biomarker Named Entity Recognition
en en.med_ner.nihss ner_nihss Named Entity Recognition
en en.extract_relation.nihss redl_nihss_biobert Relation Extraction
en en.resolve.mesh sbiobertresolve_mesh Entity Resolution
en en.resolve.mli sbiobert_base_cased_mli Embeddings
en en.resolve.ndc sbiobertresolve_ndc Entity Resolution
en en.resolve.loinc.augmented sbiobertresolve_loinc_augmented Entity Resolution
en en.resolve.clinical_snomed_procedures_measurements sbiobertresolve_clinical_snomed_procedures_measurements Entity Resolution
es es.embed.roberta_base_biomedical roberta_base_biomedical Embeddings
es es.med_ner.roberta_ner_diag_proc roberta_ner_diag_proc Named Entity Recognition
es es.resolve.snomed robertaresolve_snomed Entity Resolution
en en.med_ner.covid_trials ner_covid_trials Named Entity Recognition
en en.classify.token_bert.bionlp bert_token_classifier_ner_bionlp Named Entity Recognition
en en.classify.token_bert.cellular bert_token_classifier_ner_cellular Named Entity Recognition
en en.classify.token_bert.chemicals bert_token_classifier_ner_chemicals Named Entity Recognition
en en.resolve.rxnorm_augmented sbiobertresolve_rxnorm_augmented Entity Resolution
en en.resolve.rxnorm_augmented sbiobertresolve_rxnorm_augmented Entity Resolution
en en.resolve.rxnorm_augmented sbiobertresolve_rxnorm_augmented Entity Resolution
en en.resolve.umls_disease_syndrome sbiobertresolve_umls_disease_syndrome Entity Resolution
en en.resolve.umls_clinical_drugs sbiobertresolve_umls_clinical_drugs Entity Resolution
en en.classify.bert_sequence.question_statement_clinical bert_sequence_classifier_question_statement_clinical Text Classification

NLU Version 3.3.0

2000%+ Speedup on small data, 63 new models for 100+ Languages with 6 new supported Transformer classes including BERT, XLM-RoBERTa, alBERT, Longformer, XLnet based models, 48 NER profiling helathcare pipelines and much more in John Snow Labs NLU 3.3.0

We are incredibly excited to announce NLU 3.3.0 has been released! It comes with a up to 2000%+ speedup on small datasets, 6 new Types of Deep Learning transformer models, including RoBertaForTokenClassification,XlmRoBertaForTokenClassification,AlbertForTokenClassification,LongformerForTokenClassification,XlnetForTokenClassification,XlmRoBertaSentenceEmbeddings. In total there are 63 NLP Models 6 New Languages Supported which are Igbo, Ganda, Dholuo, Naija, Wolof,Kinyarwanda with their corresponding ISO codes ig, lg, lou, pcm, wo,rw with New SOTA XLM-RoBERTa models in Luganda, Kinyarwanda, Igbo, Hausa, and Amharic languages and 2 new Multilingual Embeddings with 100+ supported languages via XLM-Roberta are available.

On the healthcare NLP side we are glad to announce 18 new NLP for Healthcare models including NER Profiling pretrained pipelines to run 48 different Clinical NER and 21 Different Biobert Models At Once Over the Input Text New BERT-Based Deidentification NER Model, Sentence Entity Resolver Models For German Language New Spell Checker Model For Drugs , 3 New Sentence Entity Resolver Models (3-char ICD10CM, RxNorm_NDC, HCPCS) 5 New Clinical NER Models (Trained By BertForTokenClassification Approach) ,Radiology NER Model Trained On cheXpert Datasetand New UMLS Sentence Entity Resolver Models

Additionally 2 new tutorials are avaiable, NLU & Streamlit Crashcourse and NLU for Healthcare Crashcourse of every of the 50 + healthcare Domains and 200+ healthcare models

New Features and Improvements

2000%+ Speedup prediction for small datasets

NLU pipelines now predict up to 2000% faster by optimizing integration with Spark NLP’s light pipelines. NLU will configure usage of this automatically, but it can be turned off as well via multithread=False
NLU 3.3.0 Benchmark

50x faster saving of NLU Pipelines

Up to 50x faster saving Spark NLP/ NLU models and pipelines! We have improved the way we package TensorFlow SavedModel while saving Spark NLP models & pipelines. For instance, it used to take up to 10 minutes to save the xlm_roberta_base model before Spark NLP 3.3.0, and now it only takes up to 15 seconds!

New Annotator Classes Integrated

The following new transformer classes are available with various pretrained weights in 1 line of code :

New Transformer Models

The following models are available from the amazing Spark NLP 3.3.0 and 3.3.1 releases which includes NLP models for Yiddish, Ukrainian, Telugu, Tamil, Somali, Sindhi, Russian, Punjabi, Nepali, Marathi, Malayalam, Kannada, Indonesian, Gujrati, Bosnian, Igbo, Ganda, Dholuo, Naija, Wolof,Kinyarwanda

Language NLU Reference Spark NLP Reference Task  
ig ig.embed.xlm_roberta xlm_roberta_base_finetuned_igbo Embeddings  
ig ig.embed_sentence.xlm_roberta sent_xlm_roberta_base_finetuned_igbo Embeddings  
lg lg.embed.xlm_roberta xlm_roberta_base_finetuned_luganda Embeddings  
lg lg.embed_sentence.xlm_roberta sent_xlm_roberta_base_finetuned_luganda Embeddings  
wo wo.embed_sentence.xlm_roberta sent_xlm_roberta_base_finetuned_wolof Embeddings  
wo wo.embed.xlm_roberta xlm_roberta_base_finetuned_wolof Embeddings  
rw rw.embed_sentence.xlm_roberta sent_xlm_roberta_base_finetuned_kinyarwanda Embeddings  
rw rw.embed.xlm_roberta xlm_roberta_base_finetuned_kinyarwanda Embeddings  
sw sw.embed_sentence.xlm_roberta sent_xlm_roberta_base_finetuned_swahili Embeddings  
sw sw.embed.xlm_roberta xlm_roberta_base_finetuned_swahili Embeddings  
ha ha.embed.xlm_roberta xlm_roberta_base_finetuned_hausa Embeddings  
ha ha.embed_sentence.xlm_roberta sent_xlm_roberta_base_finetuned_hausa Embeddings  
am am.embed.xlm_roberta xlm_roberta_base_finetuned_amharic Embeddings  
am am.embed_sentence.xlm_roberta sent_xlm_roberta_base_finetuned_amharic Embeddings  
yo yo.embed_sentence.xlm_roberta sent_xlm_roberta_base_finetuned_yoruba Embeddings  
yo yo.embed.xlm_roberta xlm_roberta_base_finetuned_yoruba Embeddings  
fa fa.classify.token_roberta_token_classifier_zwnj_base_ner roberta_token_classifier_zwnj_base_ner Named Entity Recognition  
yi detect_sentence sentence_detector_dl Sentence Detection  
uk detect_sentence sentence_detector_dl Sentence Detection  
te detect_sentence sentence_detector_dl Sentence Detection  
ta detect_sentence sentence_detector_dl Sentence Detection  
so detect_sentence sentence_detector_dl Sentence Detection  
sd detect_sentence sentence_detector_dl Sentence Detection  
ru detect_sentence sentence_detector_dl Sentence Detection  
pa detect_sentence sentence_detector_dl Sentence Detection  
ne detect_sentence sentence_detector_dl Sentence Detection  
mr detect_sentence sentence_detector_dl Sentence Detection  
ml detect_sentence sentence_detector_dl Sentence Detection  
kn detect_sentence sentence_detector_dl Sentence Detection  
id detect_sentence sentence_detector_dl Sentence Detection  
gu detect_sentence sentence_detector_dl Sentence Detection  
bs detect_sentence sentence_detector_dl Sentence Detection  
en en.classify.token_roberta_large_token_classifier_conll03 roberta_large_token_classifier_conll03 Named Entity Recognition  
en en.classify.token_roberta_base_token_classifier_ontonotes roberta_base_token_classifier_ontonotes Named Entity Recognition  
en en.classify.token_roberta_base_token_classifier_conll03 roberta_base_token_classifier_conll03 Named Entity Recognition  
en en.classify.token_distilroberta_base_token_classifier_ontonotes distilroberta_base_token_classifier_ontonotes Named Entity Recognition  
en en.classify.token_albert_large_token_classifier_conll03 albert_large_token_classifier_conll03 Named Entity Recognition  
en en.classify.token_albert_base_token_classifier_conll03 albert_base_token_classifier_conll03 Named Entity Recognition  
en en.classify.token_xlnet_base_token_classifier_conll03 xlnet_base_token_classifier_conll03 Named Entity Recognition  
en en.classify.token_roberta.large_token_classifier_ontonotes roberta_large_token_classifier_ontonotes Named Entity Recognition  
en en.classify.token_albert.xlarge_token_classifier_conll03 albert_xlarge_token_classifier_conll03 Named Entity Recognition  
en en.classify.token_xlnet.large_token_classifier_conll03 xlnet_large_token_classifier_conll03 Named Entity Recognition  
en en.classify.token_longformer.base_token_classifier_conll03 longformer_base_token_classifier_conll03 Named Entity Recognition  
xx xx.classify.token_xlm_roberta.token_classifier_ner_40_lang xlm_roberta_token_classifier_ner_40_lang Named Entity Recognition  
xx xx.embed.xlm_roberta_large xlm_roberta_large Embeddings  

New Healthcare models

The following models are available from the amazing Spark NLP for Healthcare releases 3.3.0, 3.2.3, 3.3.1, which includes 48 Multi-NER tuning pipelines, BERT-based DEidentification, German NER resolvers, Spell Checkers for Drugs, 5 ner NER models trained via BErtForTokenClassification, NER models for Radiology CID10CM, RxNORM NDC and HCPCSS models and UMLS sentence resolver models

Language NLU Reference Spark NLP Reference Task
de de.resolve.snomed sbertresolve_snomed Entity Resolution
de de.resolve.icd10gm sbertresolve_icd10gm Entity Resolution
en en.med_ner.profiling_clinical ner_profiling_clinical Pipeline Healthcare
en en.med_ner.profiling_biobert ner_profiling_biobert Pipeline Healthcare
en en.med_ner.chexpert ner_chexpert Named Entity Recognition
en en.classify.token_bert.ner_bacteria bert_token_classifier_ner_bacteria Named Entity Recognition
en en.classify.token_bert.ner_anatomy bert_token_classifier_ner_anatomy Named Entity Recognition
en en.classify.token_bert.ner_drugs bert_token_classifier_ner_drugs Named Entity Recognition
en en.classify.token_bert.ner_jsl_slim bert_token_classifier_ner_jsl_slim Named Entity Recognition
en en.classify.token_bert.ner_ade bert_token_classifier_ner_ade Named Entity Recognition
en en.resolve.rxnorm_ndc sbiobertresolve_rxnorm_ndc Entity Resolution
en en.resolve.icd10cm_generalised sbiobertresolve_icd10cm_generalised Entity Resolution
en en.resolve.hcpcs sbiobertresolve_hcpcs Entity Resolution
en en.spell.drug_norvig spellcheck_drug_norvig Spell Check
en en.classify.token_bert.ner_deid bert_token_classifier_ner_deid Named Entity Recognition
en en.classify.token_bert.ner_chemical bert_token_classifier_ner_chemicals Named Entity Recognition
en en.resolve.umls_disease_syndrome sbiobertresolve_umls_disease_syndrome Entity Resolution
en en.resolve.umls_clinical_drugs sbiobertresolve_umls_clinical_drugs Entity Resolution

Updated Model Names

The nlu model references have been updated to better reflect their use-cases.

  • en.classify.token_bert.conll03
  • en.classify.token_bert.large_conll03
  • en.classify.token_bert.ontonote
  • en.classify.token_bert.large_ontonote
  • en.classify.token_bert.few_nerd
  • en.classify.token_bert.classifier_ner_btc
  • es.classify.token_bert.spanish_ner
  • ja.classify.token_bert.classifier_ner_ud_gsd
  • fa.classify.token_bert.parsbert_armanner
  • fa.classify.token_bert.parsbert_ner
  • fa.classify.token_bert.parsbert_peymaner
  • sv.classify.token_bert.swedish_ner
  • tr.classify.token_bert.turkish_ner
  • en.classify.token_bert.ner_clinical
  • en.classify.token_bert.ner_jsl

New Tutorial Videos

Optional get_embeddings parameter for pipelines

NLU pipelines can now be forced to not return embeddings via get_embeddings parameter.

Updated Compatibility Docs

Added documentation section regarding compatibility of NLU, Spark NLP and Spark NLP for healthcare

Bugfixes

  • Fixed a bug with Pyspark versions 3.0 and below that caused failure of predicting with pipeline
  • Fixed a bug that caused the results of TokenClassifier Models to not be properly extracted

Additional NLU ressources

Install NLU in 1 line!

* Install NLU on Google Colab : !wget https://setup.johnsnowlabs.com/nlu/colab.sh  -O - | bash
* Install NLU on Kaggle       : !wget https://setup.johnsnowlabs.com/nlu/kaggle.sh  -O - | bash
* Install NLU via Pip         : ! pip install nlu pyspark streamlit==0.80.0`

NLU Version 3.2.1

27 new models in 7 Languages, including Japanese NER, resolution models for SNOMED, ICDO, CPT and RxNorm codes and much more in NLU 3.2.1

We are very excited to announce NLU 3.2.1! This release comes with models 27 new models for 7 languages which are transformer based.
New NER-Classifiers, BertSentenceEmbeddings, BertEmbeddings and BertForTokenClassificationEmbeddings for Japanese, German, Dutch, Swedish, Spanish, French and English.
For healthcare there are new Entity Resolvers and MedicalNerModels for Snomed Conditions, Cpt Measurements, Icd0, Rxnorm Dispositions, Posology and Deidentification. Finally, a new tutorial notebook and a webinar are available, which showcase almost every feature of NLU for the over 50 Domains in Healthcare/Clinical/Biomedical/etc..

New Transformer Models

Models in Japanese, German, Dutch, Swedish, Spanish, French and English from the great Spark NLP 3.2.3 release

nlu.load() Refrence Spark NLP Refrence Annotater class Language
en.embed.bert.base_uncased_legal bert_base_uncased_legal BertEmbeddings en
en.embed_sentence.bert.base_uncased_legal sent_bert_base_uncased_legal BertSentenceEmbeddings en
en.embed.token_bert.classifier_ner_btc bert_token_classifier_ner_btc BertForTokenClassification en
es.embed.bert.base_uncased bert_base_uncased BertEmbeddings es
es.embed.bert.base_cased bert_base_cased BertEmbeddings es
es.embed_sentence.bert.base_uncased sent_bert_base_uncased BertSentenceEmbeddings es
es.embed_sentence.bert.base_cased sent_bert_base_cased BertSentenceEmbeddings es
el.embed.bert.base_uncased bert_base_uncased BertEmbeddings el
el.embed_sentence.bert.base_uncased sent_bert_base_uncased BertSentenceEmbeddings el
sv.embed.bert.base_cased bert_base_cased BertEmbeddings sv
sv.embed_sentence.bert.base_cased sent_bert_base_cased BertSentenceEmbeddings sv
nl.embed_sentence.bert.base_cased sent_bert_base_cased BertSentenceEmbeddings nl
nl.embed.bert.base_cased bert_base_cased BertEmbeddings nl
fr.classify.sentiment.bert classifierdl_bert_sentiment ClassifierDLModel fr
ja.embed.glove.cc_300d japanese_cc_300d WordEmbeddingsModel ja
ja.ner.ud_gsd_cc_300d ner_ud_gsd_cc_300d NerDLModel ja
ja.ner.ud_gsd_xlm_roberta_base ner_ud_gsd_xlm_roberta_base NerDLModel ja
ja.embed.token_bert.classifier_ner_ud_gsd bert_token_classifier_ner_ud_gsd BertForTokenClassification ja
de.embed_sentence.bert.base_cased sent_bert_base_cased BertSentenceEmbeddings de
de.classify.sentiment.bert classifierdl_bert_sentiment ClassifierDLModel de

New Healthcare Transformer Models

Models for Snomed Conditions, Cpt Measurements, Icd0, Rxnorm Dispositions, Posology and Deidentification from the amazing Spark NLP 3.2.2 for Healthcare Release

nlu.load() Refrences Spark NLP Refrence Annotater class Language
en.resolve.snomed_conditions sbertresolve_snomed_conditions SentenceEntityResolverModel en
en.resolve.cpt.procedures_measurements sbiobertresolve_cpt_procedures_measurements_augmented SentenceEntityResolverModel en
en.resolve.icdo.base sbiobertresolve_icdo_base SentenceEntityResolverModel en
en.resolve.rxnorm.disposition.sbert sbertresolve_rxnorm_disposition SentenceEntityResolverModel en
en.resolve.rxnorm_disposition.sbert sbertresolve_rxnorm_disposition SentenceEntityResolverModel en
en.med_ner.posology.experimental ner_posology_experimental MedicalNerModel en
en.med_ner.deid.subentity_augmented ner_deid_subentity_augmented MedicalNerModel en

New Notebooks

Enhancements

  • Columns of the Pandas DataFrame returned by NLU will now be sorted alphabetically

Bugfixes

  • Fixed a bug that caused output levels no beeing inferred properly
  • Fixed a bug that caused SentenceResolver visualizations not to appear.

NLU Version 3.2.0

100+ Transformers Models in 40+ languages, 3-D Streamlit Entity-Embedding-Manifold visualizations, Multi-Lingual NER, Longformers, TokenDistilBERT, Trainable Sentence Resolvers, 7% less memory usage and much more in NLU 3.2.0

We are extremely excited to announce the release of NLU 3.2.0 which marks the 1-year anniversary of the birth of this magical library.
This release packs features and improvements in every division of NLU’s aspects, 89 new NLP models with new Models including Longformer, TokenBert, TokenDistilBert and Multi-Lingual NER for 40+ Languages. 12 new Healthcare models with trainable sentence resolvers and models Adverse Drug Relations, Clinical Token Bert Models, NER Models for Radiology, Drugs, Posology, Administration Cycles, RXNorm, and new Medical Assertion models. New Streamlit visualizations enable you to see Entities in 3-D, 2-D, and 1-D Manifolds which are applicable to Entities and their Embeddings, Detected by Named-Entity-Recognizer models.
Finally, a ~7% decrease in Memory consumption in NLU’s core which benefits every computation, achieved by leveraging Pyarrow. We are incredibly thankful to our community, which helped us come this far, and are looking forward to another magical year of NLU!

Streamlit Entity Manifold visualization

function pipe.viz_streamlit_entity_embed_manifold

Visualize recognized entities by NER models via their Entity Embeddings in 1-D, 2-D, or 3-D by Reducing Dimensionality via 10+ Supported methods from Manifold Algorithms and Matrix Decomposition Algorithms. You can pick additional NER models and compare them via the GUI dropdown on the left.

  • Reduces Dimensionality of high dimensional Entity Embeddings to 1-D, 2-D, or 3-D and plot the resulting data in an interactive Plotly plot
  • Applicable with any of the 330+ Named Entity Recognizer models
  • Gemerates NUM-DIMENSIONS * NUM-NER-MODELS * NUM-DIMENSION-REDUCTION-ALGOS plots
nlu.load('ner').viz_streamlit_sentence_embed_manifold(['Hello From John Snow Labs', 'Peter loves to visit New York'])

or just run

streamlit run https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/examples/streamlit/09_entity_embedding_manifolds.py

function parameters pipe.viz_streamlit_sentence_embed_manifold

| Argument | Type | Default |Description | |—————————-|————|———————————————————–|———————————————————| |default_texts| List[str] |”Donald Trump likes to visit New York”, “Angela Merkel likes to visit Berlin!”, ‘Peter hates visiting Paris’)| List of strings to apply classifiers, embeddings, and manifolds to. |
| title | str | 'NLU ❤️ Streamlit - Prototype your NLP startup in 0 lines of code🚀' | Title of the Streamlit app |sub_title| Optional[str] | “Apply any of the 10+ Manifold or Matrix Decomposition algorithms to reduce the dimensionality of Entity Embeddings to 1-D, 2-D and 3-D “ | Sub title of the Streamlit app |
|default_algos_to_apply| List[str] | ["TSNE", "PCA"] | A list Manifold and Matrix Decomposition Algorithms to apply. Can be either 'TSNE','ISOMAP','LLE','Spectral Embedding', 'MDS','PCA','SVD aka LSA','DictionaryLearning','FactorAnalysis','FastICA' or 'KernelPCA', |
|target_dimensions| List[int] | (1,2,3) | Defines the target dimension embeddings will be reduced to | |show_algo_select| bool | True | Show selector for Manifold and Matrix Decomposition Algorithms |
| set_wide_layout_CSS | bool | True | Whether to inject custom CSS or not.|
|num_cols | int | 2 | How many columns should for the layout in streamlit when rendering the similarity matrixes.|
| key | str | "NLU_streamlit" | Key for the Streamlit elements drawn | | show_logo | bool | True | Show logo | | display_infos | bool | False | Display additonal information about ISO codes and the NLU namespace structure.|
| n_jobs | Optional[int] | 3| False | How many cores to use for paralellzing when using Sklearn Dimension Reduction algorithms. |

Sentence Entity Resolver Training

Sentence Entity Resolver Training Tutorial Notebook Named Entities are sub pieces in textual data which are labeled with classes.
These classes and strings are still ambiguous though and it is not possible to group semantically identically entities without any definition of terminology. With the Sentence Resolver you can train a state-of-the-art deep learning architecture to map entities to their unique terminological representation.

Train a Sentence resolver on a dataset with columns named y , _y and text. y is a label, _y is an extra identifier label, text is the raw text

import pandas as pd 
import nlu
dataset = pd.DataFrame({
    'text': ['The Tesla company is good to invest is', 'TSLA is good to invest','TESLA INC. we should buy','PUT ALL MONEY IN TSLA inc!!'],
    'y': ['23','23','23','23'],
    '_y': ['TESLA','TESLA','TESLA','TESLA'],

})

trainable_pipe = nlu.load('train.resolve_sentence')
fitted_pipe  = trainable_pipe.fit(dataset)
res = fitted_pipe.predict(dataset)
fitted_pipe.predict(["Peter told me to buy Tesla ", 'I have money to loose, is TSLA a good option?'])
  sentence_resolution_resolve_sentence_confidence sentence_resolution_resolve_sentence_code sentence_resolution_resolve_sentence sentence
0 ‘1.0000’ ‘23’ ‘TESLA’ ‘The Tesla company is good to invest is’
1 ‘1.0000’ ‘23’ ‘TESLA’ ‘TSLA is good to invest’
2 ‘1.0000’ ‘23’ ‘TESLA’ ‘TESLA INC. we should buy’
3 ‘1.0000’ ‘23’ ‘TESLA’ ‘PUT ALL MONEY IN TSLA inc!!’

Alternatively you can also use non-default healthcare embeddings.

trainable_pipe = nlu.load('en.embed.glove.biovec train.resolve_sentence')

Transformer Models

New models from the spectacular Spark NLP 3.2.0 + releases are integrated. 89 new models in total, with new LongFormer, TokenBert, TokenDistilBert and Multi-Lingual NER for 40+ languages. The supported languages with their ISO 639-1 code are : af, ar, bg, bn, de, el, en, es, et, eu, fa, fi, fr, he, hi, hu, id, it, ja, jv, ka, kk, ko, ml, mr, ms, my, nl, pt, ru, sw, ta, te, th, tl, tr, ur, vi, yo, and zh

nlu.load() Refrence Spark NLP Refrence Annotator Class language
en.embed.longformer longformer_base_4096 LongformerEmbeddings en
en.embed.longformer.large longformer_large_4096 LongformerEmbeddings en
en.ner.ontonotes_roberta_base ner_ontonotes_roberta_base NerDLModel en
en.ner.ontonotes_roberta_large ner_ontonotes_roberta_large NerDLModel en
en.ner.ontonotes_distilbert_base_cased ner_ontonotes_distilbert_base_cased NerDLModel en
en.ner.conll_bert_base_cased ner_conll_bert_base_cased NerDLModel en
en.ner.conll_distilbert_base_cased ner_conll_distilbert_base_cased NerDLModel en
en.ner.conll_roberta_base ner_conll_roberta_base NerDLModel en
en.ner.conll_roberta_large ner_conll_roberta_large NerDLModel en
en.ner.conll_xlm_roberta_base ner_conll_xlm_roberta_base NerDLModel en
en.ner.conll_longformer_large_4096 ner_conll_longformer_large_4096 NerDLModel en
en.embed.token_bert.conll03 bert_base_token_classifier_conll03 NerDLModel en
en.embed.token_bert.large_conll03 bert_large_token_classifier_conll03 NerDLModel en
en.embed.token_bert.ontonote bert_base_token_classifier_ontonote NerDLModel en
en.embed.token_bert.large_ontonote bert_large_token_classifier_ontonote NerDLModel en
en.embed.token_bert.few_nerd bert_base_token_classifier_few_nerd NerDLModel en
fa.embed.token_bert.parsbert_armanner bert_token_classifier_parsbert_armanner NerDLModel fa
fa.embed.token_bert.parsbert_ner bert_token_classifier_parsbert_ner NerDLModel fa
fa.embed.token_bert.parsbert_peymaner bert_token_classifier_parsbert_peymaner NerDLModel fa
tr.embed.token_bert.turkish_ner bert_token_classifier_turkish_ner NerDLModel tr
es.embed.token_bert.spanish_ner bert_token_classifier_spanish_ner NerDLModel es
sv.embed.token_bert.swedish_ner bert_token_classifier_swedish_ner NerDLModel sv
en.ner.fewnerd nerdl_fewnerd_100d NerDLModel en
en.ner.fewnerd_subentity nerdl_fewnerd_subentity_100d NerDLModel en
en.ner.movie ner_mit_movie_complex_bert_base_cased NerDLModel en
en.ner.movie_complex ner_mit_movie_complex_bert_base_cased NerDLModel en
en.ner.movie_simple ner_mit_movie_complex_bert_base_cased NerDLModel en
en.ner.mit_movie_complex_bert ner_mit_movie_complex_bert_base_cased NerDLModel en
en.ner.mit_movie_complex_distilbert ner_mit_movie_complex_distilbert_base_cased NerDLModel en
en.ner.mit_movie_simple ner_mit_movie_simple_distilbert_base_cased NerDLModel en
en.embed_sentence.bert_use_cmlm_en_base sent_bert_use_cmlm_en_base BertSentenceEmbeddings en
en.embed_sentence.bert_use_cmlm_en_large sent_bert_use_cmlm_en_large BertSentenceEmbeddings en
xx.ner.xtreme_glove_840B_300 ner_xtreme_glove_840B_300 NerDLModel xx
xx.ner.xtreme_xlm_roberta_xtreme_base ner_xtreme_xlm_roberta_xtreme_base NerDLModel xx
xx.ner.wikiner_glove_840B_300 ner_wikiner_glove_840B_300 NerDLModel xx
xx.ner.wikiner_xlm_roberta_base ner_wikiner_xlm_roberta_base NerDLModel xx
xx.embed_sentence.bert_use_cmlm_multi_base_br sent_bert_use_cmlm_multi_base_br BertSentenceEmbeddings xx
xx.embed_sentence.bert_use_cmlm_multi_base sent_bert_use_cmlm_multi_base BertSentenceEmbeddings xx
xx.embed.xlm_roberta_xtreme_base xlm_roberta_xtreme_base XlmRoBertaEmbeddings xx
xx.embed.bert_base_multilingual_cased bert_base_multilingual_cased Embeddings xx
xx.embed.bert_base_multilingual_uncased bert_base_multilingual_uncased Embeddings xx
xx.af.translate_to.ru opus_tatoeba_af_ru Translation xx
xx.he.translate_to.fr opus_tatoeba_he_fr Translation xx
xx.it.translate_to.he opus_tatoeba_it_he Translation xx
xx.cs.translate_to.sv opus_mt_cs_sv Translation xx
tr.classify.cyberbullying classifierdl_berturk_cyberbullying Pipelines tr
zh.embed.xlnet chinese_xlnet_base Embeddings zh
de.classify.news classifierdl_bert_news Pipelines de
tr.classify.berturk_cyberbullying classifierdl_berturk_cyberbullying_pipeline Pipelines tr
de.classify.bert_news classifierdl_bert_news_pipeline Pipelines de
en.classify.electra_questionpair classifierdl_electra_questionpair_pipeline Pipelines en
tr.classify.bert_news classifierdl_bert_news_pipeline Pipelines tr
en.ner.conll_elmo ner_conll_elmo NerDLModel en
en.ner.conll_albert_base_uncased ner_conll_albert_base_uncased NerDLModel en
en.ner.conll_albert_large_uncased ner_conll_albert_large_uncased NerDLModel en
en.ner.conll_xlnet_base_cased ner_conll_xlnet_base_cased NerDLModel en
xx.embed.bert.muril bert_muril BertEmbeddings xx
en.embed.bert.wiki_books_sst2 bert_wiki_books_sst2 BertEmbeddings en
en.embed.bert.wiki_books_squad2 bert_wiki_books_squad2 BertEmbeddings en
en.embed.bert.wiki_books_qqp bert_wiki_books_qqp BertEmbeddings en
en.embed.bert.wiki_books_qnli bert_wiki_books_qnli BertEmbeddings en
en.embed.bert.wiki_books_mnli bert_wiki_books_mnli BertEmbeddings en
en.embed.bert.wiki_books bert_wiki_books BertEmbeddings en
en.embed.bert.pubmed_squad2 bert_pubmed_squad2 BertEmbeddings en
en.embed.bert.pubmed bert_pubmed BertEmbeddings en
en.embed_sentence.bert.wiki_books_sst2 sent_bert_wiki_books_sst2 BertSentenceEmbeddings en
en.embed_sentence.bert.wiki_books_squad2 sent_bert_wiki_books_squad2 BertSentenceEmbeddings en
en.embed_sentence.bert.wiki_books_qqp sent_bert_wiki_books_qqp BertSentenceEmbeddings en
en.embed_sentence.bert.wiki_books_qnli sent_bert_wiki_books_qnli BertSentenceEmbeddings en
en.embed_sentence.bert.wiki_books_mnli sent_bert_wiki_books_mnli BertSentenceEmbeddings en
en.embed_sentence.bert.wiki_books sent_bert_wiki_books BertSentenceEmbeddings en
en.embed_sentence.bert.pubmed_squad2 sent_bert_pubmed_squad2 BertSentenceEmbeddings en
en.embed_sentence.bert.pubmed sent_bert_pubmed BertSentenceEmbeddings en
xx.embed_sentence.bert.muril sent_bert_muril BertSentenceEmbeddings xx
yi.detect_sentence sentence_detector_dl SentenceDetectorDLModel yi
uk.detect_sentence sentence_detector_dl SentenceDetectorDLModel uk
te.detect_sentence sentence_detector_dl SentenceDetectorDLModel te
ta.detect_sentence sentence_detector_dl SentenceDetectorDLModel ta
so.detect_sentence sentence_detector_dl SentenceDetectorDLModel so
sd.detect_sentence sentence_detector_dl SentenceDetectorDLModel sd
ru.detect_sentence sentence_detector_dl SentenceDetectorDLModel ru
pa.detect_sentence sentence_detector_dl SentenceDetectorDLModel pa
ne.detect_sentence sentence_detector_dl SentenceDetectorDLModel ne
mr.detect_sentence sentence_detector_dl SentenceDetectorDLModel mr
ml.detect_sentence sentence_detector_dl SentenceDetectorDLModel ml
kn.detect_sentence sentence_detector_dl SentenceDetectorDLModel kn
bs.detect_sentence sentence_detector_dl SentenceDetectorDLModel bs
id.detect_sentence sentence_detector_dl SentenceDetectorDLModel id
gu.detect_sentence sentence_detector_dl SentenceDetectorDLModel gu

New Healthcare Transformer Models

12 new models from the amazing Spark NLP for Healthcare 3.2.0+ releases, including models for genetic variants, radiology, assertion, rxnorm, adverse drugs and new clinical tokenbert models that improves accuracy by 4% compared to the previous models.

nlu.load() Refrence Spark NLP Refrence Annotator Class
en.med_ner.radiology.wip_greedy_biobert jsl_rd_ner_wip_greedy_biobert MedicalNerModel
en.med_ner.genetic_variants ner_genetic_variants MedicalNerModel
en.med_ner.jsl_slim ner_jsl_slim MedicalNerModel
en.med_ner.jsl_greedy_biobert ner_jsl_greedy_biobert MedicalNerModel
en.embed.token_bert.ner_clinical bert_token_classifier_ner_clinical MedicalNerModel
en.embed.token_bert.ner_jsl bert_token_classifier_ner_jsl MedicalNerModel
en.relation.ade redl_ade_biobert RelationExtractionDLModel
en.relation.ade_clinical re_ade_clinical RelationExtractionDLModel
en.relation.ade_biobert re_ade_biobert RelationExtractionDLModel
en.resolve.rxnorm_disposition sbiobertresolve_rxnorm_disposition SentenceEntityResolverModel
en.assert.jsl assertion_jsl AssertionDLModel
en.assert.jsl_large assertion_jsl_large AssertionDLModel

PyArrow Memory Optimizations

Optimized integration with Pyarrow to share memory between the Python Virtual Machine and Java Virtual Machine which yields around 7% less memory consumption on average in all computations. This improvement will take effect for everyone using the default pyspark installation, which comes with a compatible Pyarrow Version.
If you manually install or upgrade Pyarrow, please refer to the official Spark docs and make sure you have a Pyarrow version installed that works with your Pyspark version. Memory Benchmark

New Notebooks

Bugfixes

  • Fixed a bug that caused the similarity matrix calculations to generate NaNs and crash

Additional NLU ressources

Install NLU in 1 line!

* Install NLU on Google Colab : !wget https://setup.johnsnowlabs.com/nlu/colab.sh  -O - | bash
* Install NLU on Kaggle       : !wget https://setup.johnsnowlabs.com/nlu/kaggle.sh  -O - | bash
* Install NLU via Pip         : ! pip install nlu pyspark streamlit==0.80.0`

NLU Version 3.1.1

Sentence Embedding Visualizations, 20+ New Models, 2 New Trainable Models, Drug Normalizer and more in John Snow Labs NLU 3.1.1

We are very excited to announce NLU 3.1.1 has been released!
It features a new Sentence Embedding visualization component for Streamlit which supports all 10+ previous dimension reduction techniques. Additionally, all embedding visualizations now support Latent Dirichlet Allocation for dimension reduction. Finally, 2 new trainable models for NER and chunk resolution are supported, a new drug normalizer algorithm has been added, 20+ new pre-trained models including Multi-Lingual, German, various healthcare models and improved NER defaults when using licensed models that have NER dependencies.

Streamlit Sentence Embedding visualization via Manifold and Matrix Decomposition algorithms

function pipe.viz_streamlit_sentence_embed_manifold

Visualize Sentence Embeddings in 1-D, 2-D, or 3-D by Reducing Dimensionality via 12 Supported methods from Manifold Algorithms and Matrix Decomposition Algorithms. Additionally, you can color the lower dimensional points with a label that has been previously assigned to the text by specifying a list of nlu references in the additional_classifiers_for_coloring parameter. You can also select additional classifiers via the GUI.

  • Reduces Dimensionality of high dimensional Sentence Embeddings to 1-D, 2-D, or 3-D and plot the resulting data in an interactive Plotly plot
  • Applicable with any of the 100+ Sentence Embedding models
  • Color points by classifying with any of the 100+ Document Classifiers
  • Gemerates NUM-DIMENSIONS * NUM-EMBEDDINGS * NUM-DIMENSION-REDUCTION-ALGOS plots
text= """You can visualize any of the 100 + Sentence Embeddings
with 10+ dimension reduction algorithms
and view the results in 3D, 2D, and 1D  
which can be colored by various classifier labels!
"""
nlu.load('embed_sentence.bert').viz_streamlit_sentence_embed_manifold(text)

function parameters pipe.viz_streamlit_sentence_embed_manifold

| Argument | Type | Default |Description | |—————————-|————|———————————————————–|———————————————————| |default_texts| List[str] | (“Donald Trump likes to party!”, “Angela Merkel likes to party!”, ‘Peter HATES TO PARTTY!!!! :(‘) | List of strings to apply classifiers, embeddings, and manifolds to. |
| text | Optional[str] | 'Billy likes to swim' | Text to predict classes for. |
|sub_title| Optional[str] | “Apply any of the 11 Manifold or Matrix Decomposition algorithms to reduce the dimensionality of Sentence Embeddings to 1-D, 2-D and 3-D “ | Sub title of the Streamlit app |
|default_algos_to_apply| List[str] | ["TSNE", "PCA"] | A list Manifold and Matrix Decomposition Algorithms to apply. Can be either 'TSNE','ISOMAP','LLE','Spectral Embedding', 'MDS','PCA','SVD aka LSA','DictionaryLearning','FactorAnalysis','FastICA' or 'KernelPCA', |
|target_dimensions| List[int] | (1,2,3) | Defines the target dimension embeddings will be reduced to | |show_algo_select| bool | True | Show selector for Manifold and Matrix Decomposition Algorithms |
|show_embed_select| bool | True | Show selector for Embedding Selection |
|show_color_select| bool | True | Show selector for coloring plots | |display_embed_information | bool | True | Show additional embedding information like dimension, nlu_reference, spark_nlp_reference, sotrage_reference, modelhub link and more.|
| set_wide_layout_CSS | bool | True | Whether to inject custom CSS or not.|
|num_cols | int | 2 | How many columns should for the layout in streamlit when rendering the similarity matrixes.|
| key | str | "NLU_streamlit" | Key for the Streamlit elements drawn | |additional_classifiers_for_coloring | List[str]|['sentiment.imdb'] | List of additional NLU references to load for generting hue colors | | show_model_select | bool | True | Show a model selection dropdowns that makes any of the 1000+ models avaiable in 1 click | | model_select_position | str | 'side' | Whether to output the positions of predictions or not, see pipe.predict(positions=true) for more info |
| show_logo | bool | True | Show logo | | display_infos | bool | False | Display additonal information about ISO codes and the NLU namespace structure.|
| n_jobs | Optional[int] | 3| False | How many cores to use for paralellzing when using Sklearn Dimension Reduction algorithms. |

General Streamlit enhancements

Support for Latent Dirichlet Allocation

The Latent Dirichlet Allocation algorithm is now supported for the Word Embedding Visualizations and the Sentence Embedding Visualizations

Normalization of Vectors before calculating sentence similarity.

WordEmbedding vectors will now be normalized before calculating similarity scores, which bounds each similarity between 0 and 1

Control order of plots

You can now control the order in Which visualizations appear in the main GUI

Sentence Embedding Visualization

Chunk Entity Resolver Training

Chunk Entity Resolver Training Tutorial Notebook Named Entities are sub pieces in textual data which are labeled with classes.
These classes and strings are still ambigous though and it is not possible to group semantically identically entities without any definition of terminology. With the Chunk Resolver you can train a state-of-the-art deep learning architecture to map entities to their unique terminological representation.

Train a chunk resolver on a dataset with columns named y , _y and text. y is a label, _y is an extra identifier label, text is the raw text

import pandas as pd 
dataset = pd.DataFrame({
    'text': ['The Tesla company is good to invest is', 'TSLA is good to invest','TESLA INC. we should buy','PUT ALL MONEY IN TSLA inc!!'],
    'y': ['23','23','23','23']
    '_y': ['TESLA','TESLA','TESLA','TESLA'], 

})


trainable_pipe = nlu.load('train.resolve_chunks')
fitted_pipe  = trainable_pipe.fit(dataset)
res = fitted_pipe.predict(dataset)
fitted_pipe.predict(["Peter told me to buy Tesla ", 'I have money to loose, is TSLA a good option?'])
entity_resolution_confidence entity_resolution_code entity_resolution document
‘1.0000’ ‘23’ ‘TESLA’ Peter told me to buy Tesla
‘1.0000’ ‘23’ ‘TESLA’ I have money to loose, is TSLA a good option?

Train with default glove embeddings

untrained_chunk_resolver = nlu.load('train.resolve_chunks')
trained_chunk_resolver  =  untrained_chunk_resolver.fit(df)
trained_chunk_resolver.predict(df)

Train with custom embeddings

# Use BIo GLove
untrained_chunk_resolver = nlu.load('en.embed.glove.biovec train.resolve_chunks')
trained_chunk_resolver  =  untrained_chunk_resolver.fit(df)
trained_chunk_resolver.predict(df)

Rule based NER with Context Matcher

Rule based NER with context matching tutorial notebook
Define a rule-based NER algorithm by providing Regex Patterns and resolution mappings. The confidence value is computed using a heuristic approach based on how many matches it has.
A dictionary can be provided with setDictionary to map extracted entities to a unified representation. The first column of the dictionary file should be the representation with the following columns the possible matches.

import nlu
import json
# Define helper functions to write NER rules to file 
"""Generate json with dict contexts at target path"""
def dump_dict_to_json_file(dict, path): 
  with open(path, 'w') as f: json.dump(dict, f)

"""Dump raw text file """
def dump_file_to_csv(data,path):
  with open(path, 'w') as f:f.write(data)
sample_text = """A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years prior to presentation and subsequent type two diabetes mellitus ( T2DM ), one prior episode of HTG-induced pancreatitis three years prior to presentation , associated with an acute hepatitis , and obesity with a body mass index ( BMI ) of 33.5 kg/m2 , presented with a one-week history of polyuria , polydipsia , poor appetite , and vomiting. Two weeks prior to presentation , she was treated with a five-day course of amoxicillin for a respiratory tract infection . She was on metformin , glipizide , and dapagliflozin for T2DM and atorvastatin and gemfibrozil for HTG . She had been on dapagliflozin for six months at the time of presentation . Physical examination on presentation was significant for dry oral mucosa ; significantly , her abdominal examination was benign with no tenderness , guarding , or rigidity . Pertinent laboratory findings on admission were : serum glucose 111 mg/dl , bicarbonate 18 mmol/l , anion gap 20 , creatinine 0.4 mg/dL , triglycerides 508 mg/dL , total cholesterol 122 mg/dL , glycated hemoglobin ( HbA1c ) 10% , and venous pH 7.27 . Serum lipase was normal at 43 U/L . Serum acetone levels could not be assessed as blood samples kept hemolyzing due to significant lipemia . The patient was initially admitted for starvation ketosis , as she reported poor oral intake for three days prior to admission . However , serum chemistry obtained six hours after presentation revealed her glucose was 186 mg/dL , the anion gap was still elevated at 21 , serum bicarbonate was 16 mmol/L , triglyceride level peaked at 2050 mg/dL , and lipase was 52 U/L . β-hydroxybutyrate level was obtained and found to be elevated at 5.29 mmol/L - the original sample was centrifuged and the chylomicron layer removed prior to analysis due to interference from turbidity caused by lipemia again . The patient was treated with an insulin drip for euDKA and HTG with a reduction in the anion gap to 13 and triglycerides to 1400 mg/dL , within 24 hours . Twenty days ago. Her euDKA was thought to be precipitated by her respiratory tract infection in the setting of SGLT2 inhibitor use . At birth the typical boy is growing slightly faster than the typical girl, but the velocities become equal at about seven months, and then the girl grows faster until four years. From then until adolescence no differences in velocity can be detected. 21-02-2020 21/04/2020 """

# Define Gender NER matching rules
gender_rules = {
    "entity": "Gender",
    "ruleScope": "sentence",
    "completeMatchRegex": "true"    }

# Define dict data in csv format
gender_data = '''male,man,male,boy,gentleman,he,him
female,woman,female,girl,lady,old-lady,she,her
neutral,neutral'''

# Dump configs to file 
dump_dict_to_json_file(gender_data, 'gender.csv')
dump_dict_to_json_file(gender_rules, 'gender.json')
gender_NER_pipe = nlu.load('match.context')
gender_NER_pipe.print_info()
gender_NER_pipe['context_matcher'].setJsonPath('gender.json')
gender_NER_pipe['context_matcher'].setDictionary('gender.csv', options={"delimiter":","})
gender_NER_pipe.predict(sample_text)
context_match context_match_confidence
female 0.13
she 0.13
she 0.13
she 0.13
she 0.13
boy 0.13
girl 0.13
girl 0.13

Context Matcher Parameters

You can define the following parameters in your rules.json file to define the entities to be matched

Parameter Type Description
entity str The name of this rule
regex Optional[str] Regex Pattern to extract candidates
contextLength Optional[int] defines the maximum distance a prefix and suffix words can be away from the word to match,whereas context are words that must be immediately after or before the word to match
prefix Optional[List[str]] Words preceding the regex match, that are at most contextLength characters aways
regexPrefix Optional[str] RegexPattern of words preceding the regex match, that are at most contextLength characters aways
suffix Optional[List[str]] Words following the regex match, that are at most contextLength characters aways
regexSuffix Optional[str] RegexPattern of words following the regex match, that are at most contextLength distance aways
context Optional[List[str]] list of words that must be immediatly before/after a match
contextException Optional[List[str]] ?? List of words that may not be immediatly before/after a match
exceptionDistance Optional[int] Distance exceptions must be away from a match
regexContextException Optional[str] Regex Pattern of exceptions that may not be within exceptionDistance range of the match
matchScope Optional[str] Either token or sub-token to match on character basis
completeMatchRegex Optional[str] Wether to use complete or partial matching, either "true" or "false"
ruleScope str currently only sentence supported

Drug Normalizer

Drug Normalizer tutorial notebook

Normalize raw text from clinical documents, e.g. scraped web pages or xml documents. Removes all dirty characters from text following one or more input regex patterns. Can apply unwanted character removal which a specific policy. Can apply lower case normalization.

Parameters are

  • lowercase: whether to convert strings to lowercase. Default is False.
  • policy: rule to remove patterns from text. Valid policy values are: all abbreviations, dosages Defaults is all. abbreviation policy used to expend common drugs abbreviations, dosages policy used to convert drugs dosages and values to the standard form (see examples below).
data = ["Agnogenic one half cup","adalimumab 54.5 + 43.2 gm","aspirin 10 meq/ 5 ml oral sol","interferon alfa-2b 10 million unit ( 1 ml ) injec","Sodium Chloride/Potassium Chloride 13bag"]
nlu.load('norm_drugs').predict(data)
drug_norm text
Agnogenic 0.5 oral solution Agnogenic one half cup
adalimumab 97700 mg adalimumab 54.5 + 43.2 gm
aspirin 2 meq/ml oral solution aspirin 10 meq/ 5 ml oral sol
interferon alfa - 2b 10000000 unt ( 1 ml ) injection interferon alfa-2b 10 million unit ( 1 ml ) injec
Sodium Chloride / Potassium Chloride 13 bag Sodium Chloride/Potassium Chloride 13bag

New NLU Spells

These new magical 1-liners which get new the folowing models

Open Source NLU Spells

NLU Spell Spark NLP Model
nlu.load(‘de.ner.wikiner.6B_100’) wikiner_6B_100
nlu.load(‘xx.embed.glove.glove_6B_100’) glove_6B_100

Healthcare NLU spells

NLU Spell Spark NLP Model
nlu.load(‘en.resolve.snomed_body_structure_med’) sbertresolve_snomed_bodyStructure_med
nlu.load(‘en.resolve.snomed_body_structure’) sbiobertresolve_snomed_bodyStructure
nlu.load(‘en.resolve.icdo_augmented’) sbiobertresolve_icdo_augmented
nlu.load(‘en.embed_sentence.biobert.jsl_cased’) sbiobert_jsl_cased
nlu.load(‘en.embed_sentence.biobert.jsl_umls_cased’) sbiobert_jsl_umls_cased
nlu.load(‘en.embed_sentence.bert.jsl_medium_uncased’) sbert_jsl_medium_uncased
nlu.load(‘en.embed_sentence.bert.jsl_medium_umls_uncased’) sbert_jsl_medium_umls_uncased
nlu.load(‘en.embed_sentence.bert.jsl_mini_uncased’) sbert_jsl_mini_uncased
nlu.load(‘en.embed_sentence.bert.jsl_mini_umlsuncased’) sbert_jsl_mini_umls_uncasedjsl_tiny_uncased
nlu.load(‘en.embed_sentence.bert.jsl_tiny_uncased’) sbert_jsl_tiny_uncased
nlu.load(‘en.embed_sentence.bert.jsl_tiny_umls_uncased’) sbert_jsl_tiny_umls_uncased
nlu.load(‘en.resolve.icd10cm.slim_billable_hcc’) sbiobertresolve_icd10cm_slim_billable_hcc
nlu.load(‘en.resolve.icd10cm.slim_billable_hcc_med’) sbertresolve_icd10cm_slim_billable_hcc_med
nlu.load(‘med_ner.deid.generic_augmented’) ner_deid_generic_augmented
nlu.load(‘med_ner.deid.subentity_augmented’) ner_deid_subentity_augmented
nlu.load(‘en.assert.radiology’) assertion_dl_radiology
nlu.load(‘en.relation.test_result_date’) re_test_result_date
nlu.load(‘en.med_ner.admission_events’) ner_events_admission_clinical
nlu.load(‘en.classify.ade.clinicalbert’) classifierdl_ade_clinicalbert
nlu.load(‘en.recognize_entities.posology’) recognize_entities_posology
nlu.load(‘en.embed_sentence.bluebert_cased_mli’) spark_name

Improved NER defaults

When loading licensed models that require a NER features like Assertion, Relation, Resolution, nlu will now use the en.med_ner model which maps to the Spark NLP model jsl_ner_wip_clinical as default. See https://nlp.johnsnowlabs.com/2021/03/31/jsl_ner_wip_clinical_en.html for more infos on this model.

New Notebooks

Additional NLU ressources

Install NLU in 1 line!

* Install NLU on Google Colab : !wget https://setup.johnsnowlabs.com/nlu/colab.sh  -O - | bash
* Install NLU on Kaggle       : !wget https://setup.johnsnowlabs.com/nlu/kaggle.sh  -O - | bash
* Install NLU via Pip         : ! pip install nlu pyspark==3.0.3

NLU Version 3.1.0

2600+ New Models for 200+ Languages and 10+ Dimension Reduction Algorithms for Streamlit Word-Embedding visualizations in 3-D

We are extremely excited to announce the release of NLU 3.1 ! This is our biggest release so far and it comes with over 2600+ new models in 200+ languages, including DistilBERT, RoBERTa, and XLM-RoBERTa and Huggingface based Embeddings from the incredible Spark-NLP 3.1.0 release, new Streamlit Visualizations for visualizing Word Embeddings in 3-D, 2-D, and 1-D, New Healthcare pipelines for healthcare code mappings and finally confidence extraction for open source NER models. Additionally, the NLU Namespace has been renamed to the NLU Spellbook, to reflect the magicalness of each 1-liners represented by them!

Streamlit Word Embedding visualization via Manifold and Matrix Decomposition algorithms

function pipe.viz_streamlit_word_embed_manifold

Visualize Word Embeddings in 1-D, 2-D, or 3-D by Reducing Dimensionality via 11 Supported methods from Manifold Algorithms and Matrix Decomposition Algorithms. Additionally, you can color the lower dimensional points with a label that has been previously assigned to the text by specifying a list of nlu references in the additional_classifiers_for_coloring parameter.

nlu.load('bert',verbose=True).viz_streamlit_word_embed_manifold(default_texts=THE_MATRIX_ARCHITECT_SCRIPT.split('\n'),default_algos_to_apply=['TSNE'],MAX_DISPLAY_NUM=5)

function parameters pipe.viz_streamlit_word_embed_manifold

Argument Type Default Description
default_texts List[str] (“Donald Trump likes to party!”, “Angela Merkel likes to party!”, ‘Peter HATES TO PARTTY!!!! :(‘) List of strings to apply classifiers, embeddings, and manifolds to.
text Optional[str] 'Billy likes to swim' Text to predict classes for.
sub_title Optional[str] Apply any of the 11 Manifold or Matrix Decomposition algorithms to reduce the dimensionality of Word Embeddings to 1-D, 2-D and 3-D Sub title of the Streamlit app
default_algos_to_apply List[str] ["TSNE", "PCA"] A list Manifold and Matrix Decomposition Algorithms to apply. Can be either 'TSNE','ISOMAP','LLE','Spectral Embedding', 'MDS','PCA','SVD aka LSA','DictionaryLearning','FactorAnalysis','FastICA' or 'KernelPCA'
target_dimensions List[int] (1,2,3) Defines the target dimension embeddings will be reduced to
show_algo_select bool True Show selector for Manifold and Matrix Decomposition Algorithms
show_embed_select bool True Show selector for Embedding Selection
show_color_select bool True Show selector for coloring plots
MAX_DISPLAY_NUM int 100 Cap maximum number of Tokens displayed
display_embed_information bool True Show additional embedding information like dimension, nlu_reference, spark_nlp_reference, sotrage_reference, modelhub link and more.
set_wide_layout_CSS bool True Whether to inject custom CSS or not.
num_cols int 2 How many columns should for the layout in streamlit when rendering the similarity matrixes.
key str "NLU_streamlit" Key for the Streamlit elements drawn
additional_classifiers_for_coloring List[str] ['pos', 'sentiment.imdb'] List of additional NLU references to load for generting hue colors
show_model_select bool True Show a model selection dropdowns that makes any of the 1000+ models avaiable in 1 click
model_select_position str 'side' Whether to output the positions of predictions or not, see pipe.predict(positions=true) for more info
show_logo bool True Show logo
display_infos bool False Display additonal information about ISO codes and the NLU namespace structure.
n_jobs Optional[int] False How many cores to use for paralellzing when using Sklearn Dimension Reduction algorithms.

Larger Example showcasing more dimension reduction techniques on a larger corpus :

Supported Manifold Algorithms

Supported Matrix Decomposition Algorithms

New Healthcare Pipelines

Five new healthcare code mapping pipelines:

  • nlu.load(en.resolve.icd10cm.umls): This pretrained pipeline maps ICD10CM codes to UMLS codes without using any text data. You’ll just feed white space-delimited ICD10CM codes and it will return the corresponding UMLS codes as a list. If there is no mapping, the original code is returned with no mapping.

{'icd10cm': ['M89.50', 'R82.2', 'R09.01'],'umls': ['C4721411', 'C0159076', 'C0004044']}

  • nlu.load(en.resolve.mesh.umls): This pretrained pipeline maps MeSH codes to UMLS codes without using any text data. You’ll just feed white space-delimited MeSH codes and it will return the corresponding UMLS codes as a list. If there is no mapping, the original code is returned with no mapping.

{'mesh': ['C028491', 'D019326', 'C579867'],'umls': ['C0970275', 'C0886627', 'C3696376']}

  • nlu.load(en.resolve.rxnorm.umls): This pretrained pipeline maps RxNorm codes to UMLS codes without using any text data. You’ll just feed white space-delimited RxNorm codes and it will return the corresponding UMLS codes as a list. If there is no mapping, the original code is returned with no mapping.

{'rxnorm': ['1161611', '315677', '343663'],'umls': ['C3215948', 'C0984912', 'C1146501']}

  • nlu.load(en.resolve.rxnorm.mesh): This pretrained pipeline maps RxNorm codes to MeSH codes without using any text data. You’ll just feed white space-delimited RxNorm codes and it will return the corresponding MeSH codes as a list. If there is no mapping, the original code is returned with no mapping.

{'rxnorm': ['1191', '6809', '47613'],'mesh': ['D001241', 'D008687', 'D019355']}

  • nlu.load(en.resolve.snomed.umls): This pretrained pipeline maps SNOMED codes to UMLS codes without using any text data. You’ll just feed white space-delimited SNOMED codes and it will return the corresponding UMLS codes as a list. If there is no mapping, the original code is returned with no mapping. {'snomed': ['733187009', '449433008', '51264003'],'umls': ['C4546029', 'C3164619', 'C0271267']}

In the following table the NLU and Spark-NLP references are listed:

NLU Reference Spark NLP Reference
en.resolve.icd10cm.umls icd10cm_umls_mapping
en.resolve.mesh.umls mesh_umls_mapping
en.resolve.rxnorm.umls rxnorm_umls_mapping
en.resolve.rxnorm.mesh rxnorm_mesh_mapping
en.resolve.snomed.umls snomed_umls_mapping
en.explain_doc.carp explain_clinical_doc_carp
en.explain_doc.era explain_clinical_doc_era

New Open Source Models and Pipelines

nlu.load() Refrence Spark NLP Refrence
en.embed.distilbert distilbert_base_cased
en.embed.distilbert.base distilbert_base_cased
en.embed.distilbert.base.uncased distilbert_base_uncased
en.embed.distilroberta distilroberta_base
en.embed.roberta roberta_base
en.embed.roberta.base roberta_base
en.embed.roberta.large roberta_large
xx.marian opus_mt_en_fr
xx.embed.distilbert. distilbert_base_multilingual_cased
xx.embed.xlm xlm_roberta_base
xx.embed.xlm.base xlm_roberta_base
xx.embed.xlm.twitter twitter_xlm_roberta_base
zh.embed.bert bert_base_chinese
zh.embed.bert.wwm chinese_bert_wwm
de.embed.bert bert_base_german_cased
de.embed.bert.uncased bert_base_german_uncased
nl.embed.bert bert_base_dutch_cased
it.embed.bert bert_base_italian_cased
tr.embed.bert bert_base_turkish_cased
tr.embed.bert.uncased bert_base_turkish_uncased
xx.fr.marian.translate_to.bcl opus_mt_bcl_fr
xx.tr.marian.translate_to.ar opus_mt_ar_tr
xx.sv.marian.translate_to.af opus_mt_af_sv
xx.de.marian.translate_to.ar opus_mt_ar_de
xx.fr.marian.translate_to.bi opus_mt_bi_fr
xx.es.marian.translate_to.bi opus_mt_bi_es
xx.fi.marian.translate_to.af opus_mt_af_fi
xx.fi.marian.translate_to.crs opus_mt_crs_fi
xx.fi.marian.translate_to.bem opus_mt_bem_fi
xx.sv.marian.translate_to.bem opus_mt_bem_sv
xx.it.marian.translate_to.ca opus_mt_ca_it
xx.fr.marian.translate_to.ca opus_mt_ca_fr
xx.es.marian.translate_to.bcl opus_mt_bcl_es
xx.uk.marian.translate_to.ca opus_mt_ca_uk
xx.fr.marian.translate_to.bem opus_mt_bem_fr
xx.de.marian.translate_to.af opus_mt_af_de
xx.nl.marian.translate_to.af opus_mt_af_nl
xx.fr.marian.translate_to.ase opus_mt_ase_fr
xx.es.marian.translate_to.az opus_mt_az_es
xx.es.marian.translate_to.chk opus_mt_chk_es
xx.sv.marian.translate_to.ceb opus_mt_ceb_sv
xx.es.marian.translate_to.ceb opus_mt_ceb_es
xx.es.marian.translate_to.aed opus_mt_aed_es
xx.pl.marian.translate_to.ar opus_mt_ar_pl
xx.es.marian.translate_to.bem opus_mt_bem_es
xx.eo.marian.translate_to.af opus_mt_af_eo
xx.fr.marian.translate_to.cs opus_mt_cs_fr
xx.fi.marian.translate_to.bcl opus_mt_bcl_fi
xx.es.marian.translate_to.crs opus_mt_crs_es
xx.sv.marian.translate_to.bi opus_mt_bi_sv
xx.de.marian.translate_to.bg opus_mt_bg_de
xx.ru.marian.translate_to.ar opus_mt_ar_ru
xx.es.marian.translate_to.bg opus_mt_bg_es
xx.uk.marian.translate_to.cs opus_mt_cs_uk
xx.sv.marian.translate_to.bzs opus_mt_bzs_sv
xx.es.marian.translate_to.be opus_mt_be_es
xx.es.marian.translate_to.bzs opus_mt_bzs_es
xx.fr.marian.translate_to.af opus_mt_af_fr
xx.pt.marian.translate_to.ca opus_mt_ca_pt
xx.fr.marian.translate_to.chk opus_mt_chk_fr
xx.de.marian.translate_to.ase opus_mt_ase_de
xx.it.marian.translate_to.ar opus_mt_ar_it
xx.fi.marian.translate_to.ceb opus_mt_ceb_fi
xx.cpp.marian.translate_to.cpp opus_mt_cpp_cpp
xx.fr.marian.translate_to.ber opus_mt_ber_fr
xx.ru.marian.translate_to.bg opus_mt_bg_ru
xx.es.marian.translate_to.ase opus_mt_ase_es
xx.es.marian.translate_to.af opus_mt_af_es
xx.it.marian.translate_to.bg opus_mt_bg_it
xx.sv.marian.translate_to.am opus_mt_am_sv
xx.eo.marian.translate_to.ar opus_mt_ar_eo
xx.fr.marian.translate_to.ceb opus_mt_ceb_fr
xx.es.marian.translate_to.ca opus_mt_ca_es
xx.fi.marian.translate_to.bzs opus_mt_bzs_fi
xx.de.marian.translate_to.crs opus_mt_crs_de
xx.fi.marian.translate_to.cs opus_mt_cs_fi
xx.afa.marian.translate_to.afa opus_mt_afa_afa
xx.sv.marian.translate_to.bg opus_mt_bg_sv
xx.tr.marian.translate_to.bg opus_mt_bg_tr
xx.fr.marian.translate_to.crs opus_mt_crs_fr
xx.sv.marian.translate_to.ase opus_mt_ase_sv
xx.de.marian.translate_to.cs opus_mt_cs_de
xx.eo.marian.translate_to.cs opus_mt_cs_eo
xx.sv.marian.translate_to.chk opus_mt_chk_sv
xx.sv.marian.translate_to.bcl opus_mt_bcl_sv
xx.fr.marian.translate_to.ar opus_mt_ar_fr
xx.ru.marian.translate_to.af opus_mt_af_ru
xx.he.marian.translate_to.ar opus_mt_ar_he
xx.fi.marian.translate_to.bg opus_mt_bg_fi
xx.es.marian.translate_to.ber opus_mt_ber_es
xx.es.marian.translate_to.ar opus_mt_ar_es
xx.uk.marian.translate_to.bg opus_mt_bg_uk
xx.fr.marian.translate_to.bzs opus_mt_bzs_fr
xx.el.marian.translate_to.ar opus_mt_ar_el
xx.nl.marian.translate_to.ca opus_mt_ca_nl
xx.de.marian.translate_to.bcl opus_mt_bcl_de
xx.eo.marian.translate_to.bg opus_mt_bg_eo
xx.de.marian.translate_to.efi opus_mt_efi_de
xx.bzs.marian.translate_to.de opus_mt_de_bzs
xx.fj.marian.translate_to.de opus_mt_de_fj
xx.fi.marian.translate_to.da opus_mt_da_fi
xx.no.marian.translate_to.da opus_mt_da_no
xx.cs.marian.translate_to.de opus_mt_de_cs
xx.efi.marian.translate_to.de opus_mt_de_efi
xx.gil.marian.translate_to.de opus_mt_de_gil
xx.bcl.marian.translate_to.de opus_mt_de_bcl
xx.pag.marian.translate_to.de opus_mt_de_pag
xx.kg.marian.translate_to.de opus_mt_de_kg
xx.fi.marian.translate_to.efi opus_mt_efi_fi
xx.is.marian.translate_to.de opus_mt_de_is
xx.fr.marian.translate_to.da opus_mt_da_fr
xx.pl.marian.translate_to.de opus_mt_de_pl
xx.ln.marian.translate_to.de opus_mt_de_ln
xx.pap.marian.translate_to.de opus_mt_de_pap
xx.vi.marian.translate_to.de opus_mt_de_vi
xx.no.marian.translate_to.de opus_mt_de_no
xx.eo.marian.translate_to.el opus_mt_el_eo
xx.af.marian.translate_to.de opus_mt_de_af
xx.es.marian.translate_to.ee opus_mt_ee_es
xx.eo.marian.translate_to.de opus_mt_de_eo
xx.bi.marian.translate_to.de opus_mt_de_bi
xx.mt.marian.translate_to.de opus_mt_de_mt
xx.lt.marian.translate_to.de opus_mt_de_lt
xx.bg.marian.translate_to.de opus_mt_de_bg
xx.hil.marian.translate_to.de opus_mt_de_hil
xx.eu.marian.translate_to.de opus_mt_de_eu
xx.da.marian.translate_to.de opus_mt_de_da
xx.ms.marian.translate_to.de opus_mt_de_ms
xx.he.marian.translate_to.de opus_mt_de_he
xx.et.marian.translate_to.de opus_mt_de_et
xx.es.marian.translate_to.de opus_mt_de_es
xx.fr.marian.translate_to.el opus_mt_el_fr
xx.fr.marian.translate_to.ee opus_mt_ee_fr
xx.el.marian.translate_to.de opus_mt_de_el
xx.sv.marian.translate_to.el opus_mt_el_sv
xx.es.marian.translate_to.csn opus_mt_csn_es
xx.tl.marian.translate_to.de opus_mt_de_tl
xx.pon.marian.translate_to.de opus_mt_de_pon
xx.fr.marian.translate_to.efi opus_mt_efi_fr
xx.uk.marian.translate_to.de opus_mt_de_uk
xx.ar.marian.translate_to.el opus_mt_el_ar
xx.fi.marian.translate_to.el opus_mt_el_fi
xx.ig.marian.translate_to.de opus_mt_de_ig
xx.guw.marian.translate_to.de opus_mt_de_guw
xx.iso.marian.translate_to.de opus_mt_de_iso
xx.sv.marian.translate_to.efi opus_mt_efi_sv
xx.ha.marian.translate_to.de opus_mt_de_ha
xx.fr.marian.translate_to.de opus_mt_de_fr
xx.gaa.marian.translate_to.de opus_mt_de_gaa
xx.nso.marian.translate_to.de opus_mt_de_nso
xx.ht.marian.translate_to.de opus_mt_de_ht
xx.nl.marian.translate_to.de opus_mt_de_nl
xx.sv.marian.translate_to.ee opus_mt_ee_sv
xx.fi.marian.translate_to.ee opus_mt_ee_fi
xx.de.marian.translate_to.ee opus_mt_ee_de
xx.eo.marian.translate_to.da opus_mt_da_eo
xx.es.marian.translate_to.csg opus_mt_csg_es
xx.de.marian.translate_to.da opus_mt_da_de
xx.ar.marian.translate_to.de opus_mt_de_ar
xx.hu.marian.translate_to.de opus_mt_de_hu
xx.ca.marian.translate_to.de opus_mt_de_ca
xx.pis.marian.translate_to.de opus_mt_de_pis
xx.ho.marian.translate_to.de opus_mt_de_ho
xx.de.marian.translate_to.de opus_mt_de_de
xx.lua.marian.translate_to.de opus_mt_de_lua
xx.loz.marian.translate_to.de opus_mt_de_loz
xx.crs.marian.translate_to.de opus_mt_de_crs
xx.es.marian.translate_to.da opus_mt_da_es
xx.ee.marian.translate_to.de opus_mt_de_ee
xx.it.marian.translate_to.de opus_mt_de_it
xx.ilo.marian.translate_to.de opus_mt_de_ilo
xx.ny.marian.translate_to.de opus_mt_de_ny
xx.fi.marian.translate_to.de opus_mt_de_fi
xx.ase.marian.translate_to.de opus_mt_de_ase
xx.hr.marian.translate_to.de opus_mt_de_hr
xx.sl.marian.translate_to.fi opus_mt_fi_sl
xx.sk.marian.translate_to.fi opus_mt_fi_sk
xx.ru.marian.translate_to.es opus_mt_es_ru
xx.sn.marian.translate_to.fi opus_mt_fi_sn
xx.pl.marian.translate_to.eo opus_mt_eo_pl
xx.cs.marian.translate_to.es opus_mt_es_cs
xx.wls.marian.translate_to.fi opus_mt_fi_wls
xx.gaa.marian.translate_to.fi opus_mt_fi_gaa
xx.is.marian.translate_to.fi opus_mt_fi_is
xx.ha.marian.translate_to.es opus_mt_es_ha
xx.nl.marian.translate_to.es opus_mt_es_nl
xx.ha.marian.translate_to.fi opus_mt_fi_ha
xx.fj.marian.translate_to.fi opus_mt_fi_fj
xx.ber.marian.translate_to.es opus_mt_es_ber
xx.ho.marian.translate_to.fi opus_mt_fi_ho
xx.ny.marian.translate_to.fi opus_mt_fi_ny
xx.sl.marian.translate_to.es opus_mt_es_sl
xx.ts.marian.translate_to.fi opus_mt_fi_ts
xx.el.marian.translate_to.eo opus_mt_eo_el
xx.war.marian.translate_to.fi opus_mt_fi_war
xx.cs.marian.translate_to.fi opus_mt_fi_cs
xx.loz.marian.translate_to.es opus_mt_es_loz
xx.mk.marian.translate_to.fi opus_mt_fi_mk
xx.bg.marian.translate_to.es opus_mt_es_bg
xx.srn.marian.translate_to.fi opus_mt_fi_srn
xx.is.marian.translate_to.es opus_mt_es_is
xx.hu.marian.translate_to.eo opus_mt_eo_hu
xx.tw.marian.translate_to.fi opus_mt_fi_tw
xx.mt.marian.translate_to.fi opus_mt_fi_mt
xx.fr.marian.translate_to.es opus_mt_es_fr
xx.yo.marian.translate_to.es opus_mt_es_yo
xx.xh.marian.translate_to.fi opus_mt_fi_xh
xx.lv.marian.translate_to.fi opus_mt_fi_lv
xx.de.marian.translate_to.fi opus_mt_fi_de
xx.ve.marian.translate_to.es opus_mt_es_ve
xx.es.marian.translate_to.fi opus_mt_fi_es
xx.eo.marian.translate_to.es opus_mt_es_eo
xx.cs.marian.translate_to.eo opus_mt_eo_cs
xx.mt.marian.translate_to.es opus_mt_es_mt
xx.el.marian.translate_to.es opus_mt_es_el
xx.ee.marian.translate_to.es opus_mt_es_ee
xx.de.marian.translate_to.eu opus_mt_eu_de
xx.et.marian.translate_to.es opus_mt_es_et
xx.fi.marian.translate_to.et opus_mt_et_fi
xx.wls.marian.translate_to.es opus_mt_es_wls
xx.mg.marian.translate_to.fi opus_mt_fi_mg
xx.eu.marian.translate_to.es opus_mt_es_eu
xx.lua.marian.translate_to.es opus_mt_es_lua
xx.pon.marian.translate_to.es opus_mt_es_pon
xx.mfe.marian.translate_to.fi opus_mt_fi_mfe
xx.he.marian.translate_to.eo opus_mt_eo_he
xx.id.marian.translate_to.es opus_mt_es_id
xx.xh.marian.translate_to.es opus_mt_es_xh
xx.ar.marian.translate_to.es opus_mt_es_ar
xx.crs.marian.translate_to.es opus_mt_es_crs
xx.es.marian.translate_to.eu opus_mt_eu_es
xx.tpi.marian.translate_to.fi opus_mt_fi_tpi
xx.pis.marian.translate_to.fi opus_mt_fi_pis
xx.vi.marian.translate_to.es opus_mt_es_vi
xx.es.marian.translate_to.et opus_mt_et_es
xx.rw.marian.translate_to.fi opus_mt_fi_rw
xx.gl.marian.translate_to.es opus_mt_es_gl
xx.pt.marian.translate_to.eo opus_mt_eo_pt
xx.he.marian.translate_to.fi opus_mt_fi_he
xx.af.marian.translate_to.fi opus_mt_fi_af
xx.ru.marian.translate_to.fi opus_mt_fi_ru
xx.ve.marian.translate_to.fi opus_mt_fi_ve
xx.ca.marian.translate_to.es opus_mt_es_ca
xx.tr.marian.translate_to.fi opus_mt_fi_tr
xx.ht.marian.translate_to.fi opus_mt_fi_ht
xx.nl.marian.translate_to.fi opus_mt_fi_nl
xx.iso.marian.translate_to.fi opus_mt_fi_iso
xx.fi.marian.translate_to.es opus_mt_es_fi
xx.da.marian.translate_to.eo opus_mt_eo_da
xx.ln.marian.translate_to.es opus_mt_es_ln
xx.csn.marian.translate_to.es opus_mt_es_csn
xx.pon.marian.translate_to.fi opus_mt_fi_pon
xx.af.marian.translate_to.eo opus_mt_eo_af
xx.bzs.marian.translate_to.fi opus_mt_fi_bzs
xx.no.marian.translate_to.es opus_mt_es_no
xx.es.marian.translate_to.es opus_mt_es_es
xx.lua.marian.translate_to.fi opus_mt_fi_lua
xx.yua.marian.translate_to.es opus_mt_es_yua
xx.ru.marian.translate_to.eu opus_mt_eu_ru
xx.tpi.marian.translate_to.es opus_mt_es_tpi
xx.lue.marian.translate_to.fi opus_mt_fi_lue
xx.sv.marian.translate_to.eo opus_mt_eo_sv
xx.niu.marian.translate_to.es opus_mt_es_niu
xx.tiv.marian.translate_to.fi opus_mt_fi_tiv
xx.pag.marian.translate_to.es opus_mt_es_pag
xx.run.marian.translate_to.fi opus_mt_fi_run
xx.ty.marian.translate_to.es opus_mt_es_ty
xx.gil.marian.translate_to.es opus_mt_es_gil
xx.ln.marian.translate_to.fi opus_mt_fi_ln
xx.ty.marian.translate_to.fi opus_mt_fi_ty
xx.prl.marian.translate_to.es opus_mt_es_prl
xx.kg.marian.translate_to.es opus_mt_es_kg
xx.rw.marian.translate_to.es opus_mt_es_rw
xx.kqn.marian.translate_to.fi opus_mt_fi_kqn
xx.sq.marian.translate_to.fi opus_mt_fi_sq
xx.sw.marian.translate_to.fi opus_mt_fi_sw
xx.csg.marian.translate_to.es opus_mt_es_csg
xx.ro.marian.translate_to.es opus_mt_es_ro
xx.ee.marian.translate_to.fi opus_mt_fi_ee
xx.ilo.marian.translate_to.fi opus_mt_fi_ilo
xx.eo.marian.translate_to.fi opus_mt_fi_eo
xx.iso.marian.translate_to.es opus_mt_es_iso
xx.bem.marian.translate_to.fi opus_mt_fi_bem
xx.tn.marian.translate_to.fi opus_mt_fi_tn
xx.da.marian.translate_to.es opus_mt_es_da
xx.es.marian.translate_to.eo opus_mt_eo_es
xx.ru.marian.translate_to.eo opus_mt_eo_ru
xx.rn.marian.translate_to.es opus_mt_es_rn
xx.lt.marian.translate_to.es opus_mt_es_lt
xx.guw.marian.translate_to.es opus_mt_es_guw
xx.tvl.marian.translate_to.es opus_mt_es_tvl
xx.fr.marian.translate_to.et opus_mt_et_fr
xx.ht.marian.translate_to.es opus_mt_es_ht
xx.mos.marian.translate_to.fi opus_mt_fi_mos
xx.ase.marian.translate_to.es opus_mt_es_ase
xx.crs.marian.translate_to.fi opus_mt_fi_crs
xx.bcl.marian.translate_to.fi opus_mt_fi_bcl
xx.tvl.marian.translate_to.fi opus_mt_fi_tvl
xx.lus.marian.translate_to.fi opus_mt_fi_lus
xx.he.marian.translate_to.es opus_mt_es_he
xx.pis.marian.translate_to.es opus_mt_es_pis
xx.it.marian.translate_to.es opus_mt_es_it
xx.fi.marian.translate_to.eo opus_mt_eo_fi
xx.tw.marian.translate_to.es opus_mt_es_tw
xx.aed.marian.translate_to.es opus_mt_es_aed
xx.bzs.marian.translate_to.es opus_mt_es_bzs
xx.nso.marian.translate_to.fi opus_mt_fi_nso
xx.gaa.marian.translate_to.es opus_mt_es_gaa
xx.zai.marian.translate_to.es opus_mt_es_zai
xx.no.marian.translate_to.fi opus_mt_fi_no
xx.uk.marian.translate_to.fi opus_mt_fi_uk
xx.sg.marian.translate_to.es opus_mt_es_sg
xx.ilo.marian.translate_to.es opus_mt_es_ilo
xx.bg.marian.translate_to.eo opus_mt_eo_bg
xx.pap.marian.translate_to.fi opus_mt_fi_pap
xx.ho.marian.translate_to.es opus_mt_es_ho
xx.toi.marian.translate_to.fi opus_mt_fi_toi
xx.st.marian.translate_to.es opus_mt_es_st
xx.to.marian.translate_to.fi opus_mt_fi_to
xx.kg.marian.translate_to.fi opus_mt_fi_kg
xx.sv.marian.translate_to.fi opus_mt_fi_sv
xx.tll.marian.translate_to.fi opus_mt_fi_tll
xx.ceb.marian.translate_to.es opus_mt_es_ceb
xx.ig.marian.translate_to.es opus_mt_es_ig
xx.sv.marian.translate_to.et opus_mt_et_sv
xx.af.marian.translate_to.es opus_mt_es_af
xx.pl.marian.translate_to.es opus_mt_es_pl
xx.ro.marian.translate_to.eo opus_mt_eo_ro
xx.tn.marian.translate_to.es opus_mt_es_tn
xx.sm.marian.translate_to.fi opus_mt_fi_sm
xx.mk.marian.translate_to.es opus_mt_es_mk
xx.id.marian.translate_to.fi opus_mt_fi_id
xx.hr.marian.translate_to.fi opus_mt_fi_hr
xx.sg.marian.translate_to.fi opus_mt_fi_sg
xx.hil.marian.translate_to.fi opus_mt_fi_hil
xx.nl.marian.translate_to.eo opus_mt_eo_nl
xx.pap.marian.translate_to.es opus_mt_es_pap
xx.fr.marian.translate_to.fi opus_mt_fi_fr
xx.bi.marian.translate_to.es opus_mt_es_bi
xx.fi.marian.translate_to.fi opus_mt_fi_fi
xx.nso.marian.translate_to.es opus_mt_es_nso
xx.et.marian.translate_to.fi opus_mt_fi_et
xx.uk.marian.translate_to.es opus_mt_es_uk
xx.sh.marian.translate_to.eo opus_mt_eo_sh
xx.lu.marian.translate_to.fi opus_mt_fi_lu
xx.gil.marian.translate_to.fi opus_mt_fi_gil
xx.ro.marian.translate_to.fi opus_mt_fi_ro
xx.it.marian.translate_to.eo opus_mt_eo_it
xx.hu.marian.translate_to.fi opus_mt_fi_hu
xx.bcl.marian.translate_to.es opus_mt_es_bcl
xx.fse.marian.translate_to.fi opus_mt_fi_fse
xx.hil.marian.translate_to.es opus_mt_es_hil
xx.ig.marian.translate_to.fi opus_mt_fi_ig
xx.tl.marian.translate_to.es opus_mt_es_tl
xx.pag.marian.translate_to.fi opus_mt_fi_pag
xx.guw.marian.translate_to.fi opus_mt_fi_guw
xx.swc.marian.translate_to.es opus_mt_es_swc
xx.swc.marian.translate_to.fi opus_mt_fi_swc
xx.lg.marian.translate_to.fi opus_mt_fi_lg
xx.srn.marian.translate_to.es opus_mt_es_srn
xx.hr.marian.translate_to.es opus_mt_es_hr
xx.sm.marian.translate_to.es opus_mt_es_sm
xx.de.marian.translate_to.es opus_mt_es_de
xx.st.marian.translate_to.fi opus_mt_fi_st
xx.fr.marian.translate_to.eo opus_mt_eo_fr
xx.de.marian.translate_to.et opus_mt_et_de
xx.niu.marian.translate_to.fi opus_mt_fi_niu
xx.el.marian.translate_to.fi opus_mt_fi_el
xx.efi.marian.translate_to.fi opus_mt_fi_efi
xx.war.marian.translate_to.es opus_mt_es_war
xx.mfs.marian.translate_to.es opus_mt_es_mfs
xx.bg.marian.translate_to.fi opus_mt_fi_bg
xx.lus.marian.translate_to.es opus_mt_es_lus
xx.de.marian.translate_to.eo opus_mt_eo_de
xx.it.marian.translate_to.fi opus_mt_fi_it
xx.efi.marian.translate_to.es opus_mt_es_efi
xx.ny.marian.translate_to.es opus_mt_es_ny
xx.fj.marian.translate_to.es opus_mt_es_fj
xx.ru.marian.translate_to.et opus_mt_et_ru
xx.mh.marian.translate_to.fi opus_mt_fi_mh
xx.es.marian.translate_to.ig opus_mt_ig_es
xx.sv.marian.translate_to.hu opus_mt_hu_sv
xx.lue.marian.translate_to.fr opus_mt_fr_lue
xx.fi.marian.translate_to.ha opus_mt_ha_fi
xx.ca.marian.translate_to.it opus_mt_it_ca
xx.de.marian.translate_to.ilo opus_mt_ilo_de
xx.it.marian.translate_to.he opus_tatoeba_it_he
xx.loz.marian.translate_to.fr opus_mt_fr_loz
xx.ms.marian.translate_to.fr opus_mt_fr_ms
xx.uk.marian.translate_to.it opus_mt_it_uk
xx.gaa.marian.translate_to.fr opus_mt_fr_gaa
xx.pap.marian.translate_to.fr opus_mt_fr_pap
xx.fi.marian.translate_to.ilo opus_mt_ilo_fi
xx.lg.marian.translate_to.fr opus_mt_fr_lg
xx.it.marian.translate_to.is opus_mt_is_it
xx.ms.marian.translate_to.it opus_mt_it_ms
xx.es.marian.translate_to.fr opus_mt_fr_es
xx.ar.marian.translate_to.he opus_mt_he_ar
xx.ro.marian.translate_to.fr opus_mt_fr_ro
xx.ru.marian.translate_to.fr opus_mt_fr_ru
xx.fi.marian.translate_to.ht opus_mt_ht_fi
xx.bg.marian.translate_to.it opus_mt_it_bg
xx.mh.marian.translate_to.fr opus_mt_fr_mh
xx.to.marian.translate_to.fr opus_mt_fr_to
xx.sl.marian.translate_to.fr opus_mt_fr_sl
xx.fr.marian.translate_to.gil opus_mt_gil_fr
xx.es.marian.translate_to.hr opus_mt_hr_es
xx.ilo.marian.translate_to.fr opus_mt_fr_ilo
xx.ee.marian.translate_to.fr opus_mt_fr_ee
xx.sv.marian.translate_to.he opus_mt_he_sv
xx.fr.marian.translate_to.ha opus_mt_ha_fr
xx.gil.marian.translate_to.fr opus_mt_fr_gil
xx.fi.marian.translate_to.id opus_mt_id_fi
xx.iir.marian.translate_to.iir opus_mt_iir_iir
xx.pl.marian.translate_to.fr opus_mt_fr_pl
xx.tw.marian.translate_to.fr opus_mt_fr_tw
xx.sv.marian.translate_to.gaa opus_mt_gaa_sv
xx.ar.marian.translate_to.it opus_mt_it_ar
xx.es.marian.translate_to.gil opus_mt_gil_es
xx.ase.marian.translate_to.fr opus_mt_fr_ase
xx.fr.marian.translate_to.gaa opus_mt_gaa_fr
xx.lus.marian.translate_to.fr opus_mt_fr_lus
xx.fr.marian.translate_to.iso opus_mt_iso_fr
xx.sm.marian.translate_to.fr opus_mt_fr_sm
xx.mfe.marian.translate_to.fr opus_mt_fr_mfe
xx.af.marian.translate_to.fr opus_mt_fr_af
xx.de.marian.translate_to.ig opus_mt_ig_de
xx.es.marian.translate_to.id opus_mt_id_es
xx.kqn.marian.translate_to.fr opus_mt_fr_kqn
xx.zne.marian.translate_to.fi opus_mt_fi_zne
xx.rw.marian.translate_to.fr opus_mt_fr_rw
xx.ny.marian.translate_to.fr opus_mt_fr_ny
xx.ig.marian.translate_to.fr opus_mt_fr_ig
xx.ur.marian.translate_to.hi opus_mt_hi_ur
xx.lt.marian.translate_to.it opus_mt_it_lt
xx.srn.marian.translate_to.fr opus_mt_fr_srn
xx.tiv.marian.translate_to.fr opus_mt_fr_tiv
xx.war.marian.translate_to.fr opus_mt_fr_war
xx.fr.marian.translate_to.is opus_mt_is_fr
xx.de.marian.translate_to.gaa opus_mt_gaa_de
xx.kwy.marian.translate_to.fr opus_mt_fr_kwy
xx.sv.marian.translate_to.gil opus_mt_gil_sv
xx.hr.marian.translate_to.fr opus_mt_fr_hr
xx.fr.marian.translate_to.ig opus_mt_ig_fr
xx.sv.marian.translate_to.ht opus_mt_ht_sv
xx.de.marian.translate_to.fr opus_mt_fr_de
xx.fiu.marian.translate_to.fiu opus_mt_fiu_fiu
xx.wls.marian.translate_to.fr opus_mt_fr_wls
xx.eo.marian.translate_to.hu opus_mt_hu_eo
xx.guw.marian.translate_to.fr opus_mt_fr_guw
xx.de.marian.translate_to.is opus_mt_is_de
xx.tvl.marian.translate_to.fr opus_mt_fr_tvl
xx.zne.marian.translate_to.fr opus_mt_fr_zne
xx.ha.marian.translate_to.fr opus_mt_fr_ha
xx.fi.marian.translate_to.guw opus_mt_guw_fi
xx.es.marian.translate_to.is opus_mt_is_es
xx.sv.marian.translate_to.it opus_mt_it_sv
xx.uk.marian.translate_to.fr opus_mt_fr_uk
xx.uk.marian.translate_to.hu opus_mt_hu_uk
xx.mt.marian.translate_to.fr opus_mt_fr_mt
xx.gem.marian.translate_to.gem opus_mt_gem_gem
xx.fr.marian.translate_to.fj opus_mt_fj_fr
xx.fi.marian.translate_to.gil opus_mt_gil_fi
xx.fr.marian.translate_to.hu opus_mt_hu_fr
xx.bcl.marian.translate_to.fr opus_mt_fr_bcl
xx.gmq.marian.translate_to.gmq opus_mt_gmq_gmq
xx.kg.marian.translate_to.fr opus_mt_fr_kg
xx.sn.marian.translate_to.fr opus_mt_fr_sn
xx.bg.marian.translate_to.fr opus_mt_fr_bg
xx.fr.marian.translate_to.guw opus_mt_guw_fr
xx.ts.marian.translate_to.fr opus_mt_fr_ts
xx.pis.marian.translate_to.fr opus_mt_fr_pis
xx.bi.marian.translate_to.fr opus_mt_fr_bi
xx.ln.marian.translate_to.fr opus_mt_fr_ln
xx.de.marian.translate_to.hil opus_mt_hil_de
xx.nso.marian.translate_to.fr opus_mt_fr_nso
xx.es.marian.translate_to.iso opus_mt_iso_es
xx.crs.marian.translate_to.fr opus_mt_fr_crs
xx.niu.marian.translate_to.fr opus_mt_fr_niu
xx.fr.marian.translate_to.ht opus_mt_ht_fr
xx.fi.marian.translate_to.he opus_mt_he_fi
xx.gmw.marian.translate_to.gmw opus_mt_gmw_gmw
xx.fr.marian.translate_to.hr opus_mt_hr_fr
xx.sg.marian.translate_to.fr opus_mt_fr_sg
xx.pon.marian.translate_to.fr opus_mt_fr_pon
xx.fi.marian.translate_to.gaa opus_mt_gaa_fi
xx.pag.marian.translate_to.fr opus_mt_fr_pag
xx.fi.marian.translate_to.is opus_mt_is_fi
xx.sk.marian.translate_to.fr opus_mt_fr_sk
xx.yap.marian.translate_to.fr opus_mt_fr_yap
xx.es.marian.translate_to.ha opus_mt_ha_es
xx.no.marian.translate_to.fr opus_mt_fr_no
xx.ine.marian.translate_to.ine opus_mt_ine_ine
xx.fr.marian.translate_to.id opus_mt_id_fr
xx.bzs.marian.translate_to.fr opus_mt_fr_bzs
xx.he.marian.translate_to.fr opus_tatoeba_he_fr
xx.sv.marian.translate_to.fr opus_mt_fr_sv
xx.uk.marian.translate_to.he opus_mt_he_uk
xx.fr.marian.translate_to.it opus_mt_it_fr
xx.fi.marian.translate_to.ig opus_mt_ig_fi
xx.vi.marian.translate_to.fr opus_mt_fr_vi
xx.fi.marian.translate_to.fse opus_mt_fse_fi
xx.es.marian.translate_to.guw opus_mt_guw_es
xx.tll.marian.translate_to.fr opus_mt_fr_tll
xx.lua.marian.translate_to.fr opus_mt_fr_lua
xx.yap.marian.translate_to.fi opus_mt_fi_yap
xx.es.marian.translate_to.gaa opus_mt_gaa_es
xx.sv.marian.translate_to.ig opus_mt_ig_sv
xx.ht.marian.translate_to.fr opus_mt_fr_ht
xx.el.marian.translate_to.fr opus_mt_fr_el
xx.inc.marian.translate_to.inc opus_mt_inc_inc
xx.swc.marian.translate_to.fr opus_mt_fr_swc
xx.ar.marian.translate_to.fr opus_mt_fr_ar
xx.es.marian.translate_to.ilo opus_mt_ilo_es
xx.fi.marian.translate_to.hr opus_mt_hr_fi
xx.tpi.marian.translate_to.fr opus_mt_fr_tpi
xx.ve.marian.translate_to.fr opus_mt_fr_ve
xx.sv.marian.translate_to.guw opus_mt_guw_sv
xx.sv.marian.translate_to.iso opus_mt_iso_sv
xx.sv.marian.translate_to.is opus_mt_is_sv
xx.tum.marian.translate_to.fr opus_mt_fr_tum
xx.es.marian.translate_to.ht opus_mt_ht_es
xx.ho.marian.translate_to.fr opus_mt_fr_ho
xx.efi.marian.translate_to.fr opus_mt_fr_efi
xx.es.marian.translate_to.gl opus_mt_gl_es
xx.ru.marian.translate_to.he opus_mt_he_ru
xx.fi.marian.translate_to.hil opus_mt_hil_fi
xx.eo.marian.translate_to.he opus_mt_he_eo
xx.lu.marian.translate_to.fr opus_mt_fr_lu
xx.sv.marian.translate_to.ha opus_mt_ha_sv
xx.rnd.marian.translate_to.fr opus_mt_fr_rnd
xx.st.marian.translate_to.fr opus_mt_fr_st
xx.tl.marian.translate_to.fr opus_mt_fr_tl
xx.bem.marian.translate_to.fr opus_mt_fr_bem
xx.eo.marian.translate_to.is opus_mt_is_eo
xx.is.marian.translate_to.it opus_mt_it_is
xx.hu.marian.translate_to.fr opus_mt_fr_hu
xx.yo.marian.translate_to.fi opus_mt_fi_yo
xx.iso.marian.translate_to.fr opus_mt_fr_iso
xx.de.marian.translate_to.it opus_mt_it_de
xx.ty.marian.translate_to.fr opus_mt_fr_ty
xx.hil.marian.translate_to.fr opus_mt_fr_hil
xx.eo.marian.translate_to.it opus_mt_it_eo
xx.sv.marian.translate_to.hr opus_mt_hr_sv
xx.ber.marian.translate_to.fr opus_mt_fr_ber
xx.de.marian.translate_to.guw opus_mt_guw_de
xx.fi.marian.translate_to.hu opus_mt_hu_fi
xx.es.marian.translate_to.it opus_mt_it_es
xx.de.marian.translate_to.hu opus_mt_hu_de
xx.fj.marian.translate_to.fr opus_mt_fr_fj
xx.sv.marian.translate_to.id opus_mt_id_sv
xx.xh.marian.translate_to.fr opus_mt_fr_xh
xx.yo.marian.translate_to.fr opus_mt_fr_yo
xx.ca.marian.translate_to.fr opus_mt_fr_ca
xx.es.marian.translate_to.he opus_mt_he_es
xx.de.marian.translate_to.he opus_mt_he_de
xx.pt.marian.translate_to.gl opus_mt_gl_pt
xx.ru.marian.translate_to.hy opus_mt_hy_ru
xx.mos.marian.translate_to.fr opus_mt_fr_mos
xx.ceb.marian.translate_to.fr opus_mt_fr_ceb
xx.sh.marian.translate_to.ja opus_mt_ja_sh
xx.bg.marian.translate_to.ja opus_mt_ja_bg
xx.sv.marian.translate_to.ja opus_mt_ja_sv
xx.ru.marian.translate_to.lv opus_mt_lv_ru
xx.fr.marian.translate_to.ms opus_mt_ms_fr
xx.sv.marian.translate_to.mt opus_mt_mt_sv
xx.da.marian.translate_to.ja opus_mt_ja_da
xx.de.marian.translate_to.niu opus_mt_niu_de
xx.es.marian.translate_to.niu opus_mt_niu_es
xx.sv.marian.translate_to.lus opus_mt_lus_sv
xx.sv.marian.translate_to.lg opus_mt_lg_sv
xx.sv.marian.translate_to.pon opus_mt_pon_sv
xx.ru.marian.translate_to.lt opus_mt_lt_ru
xx.fi.marian.translate_to.lg opus_mt_lg_fi
xx.sv.marian.translate_to.kg opus_mt_kg_sv
xx.fr.marian.translate_to.nl opus_mt_nl_fr
xx.ms.marian.translate_to.ms opus_mt_ms_ms
xx.es.marian.translate_to.lg opus_mt_lg_es
xx.fr.marian.translate_to.lu opus_mt_lu_fr
xx.fr.marian.translate_to.loz opus_mt_loz_fr
xx.ca.marian.translate_to.nl opus_mt_nl_ca
xx.sv.marian.translate_to.lue opus_mt_lue_sv
xx.vi.marian.translate_to.ja opus_mt_ja_vi
xx.fr.marian.translate_to.ja opus_mt_ja_fr
xx.fi.marian.translate_to.pap opus_mt_pap_fi
xx.pl.marian.translate_to.lt opus_mt_lt_pl
xx.de.marian.translate_to.ny opus_mt_ny_de
xx.fr.marian.translate_to.lue opus_mt_lue_fr
xx.gl.marian.translate_to.pt opus_mt_pt_gl
xx.fr.marian.translate_to.pap opus_mt_pap_fr
xx.uk.marian.translate_to.pl opus_mt_pl_uk
xx.fi.marian.translate_to.niu opus_mt_niu_fi
xx.ar.marian.translate_to.ja opus_mt_ja_ar
xx.es.marian.translate_to.mh opus_mt_mh_es
xx.ar.marian.translate_to.pl opus_mt_pl_ar
xx.de.marian.translate_to.pag opus_mt_pag_de
xx.es.marian.translate_to.no opus_mt_no_es
xx.es.marian.translate_to.mfs opus_mt_mfs_es
xx.fr.marian.translate_to.pis opus_mt_pis_fr
xx.eo.marian.translate_to.pt opus_mt_pt_eo
xx.de.marian.translate_to.lt opus_mt_lt_de
xx.fr.marian.translate_to.ln opus_mt_ln_fr
xx.es.marian.translate_to.pag opus_mt_pag_es
xx.fi.marian.translate_to.nl opus_mt_nl_fi
xx.vi.marian.translate_to.it opus_mt_it_vi
xx.fi.marian.translate_to.ko opus_mt_ko_fi
xx.de.marian.translate_to.nso opus_mt_nso_de
xx.fr.marian.translate_to.niu opus_mt_niu_fr
xx.ca.marian.translate_to.pt opus_mt_pt_ca
xx.fr.marian.translate_to.kwy opus_mt_kwy_fr
xx.ru.marian.translate_to.no opus_mt_no_ru
xx.fi.marian.translate_to.pon opus_mt_pon_fi
xx.fi.marian.translate_to.lu opus_mt_lu_fi
xx.es.marian.translate_to.ko opus_mt_ko_es
xx.es.marian.translate_to.ny opus_mt_ny_es
xx.itc.marian.translate_to.itc opus_mt_itc_itc
xx.es.marian.translate_to.ja opus_mt_ja_es
xx.fr.marian.translate_to.mk opus_mt_mk_fr
xx.it.marian.translate_to.ms opus_mt_ms_it
xx.sv.marian.translate_to.lu opus_mt_lu_sv
xx.fr.marian.translate_to.nso opus_mt_nso_fr
xx.uk.marian.translate_to.pt opus_mt_pt_uk
xx.no.marian.translate_to.no opus_mt_no_no
xx.sv.marian.translate_to.lua opus_mt_lua_sv
xx.es.marian.translate_to.pl opus_mt_pl_es
xx.es.marian.translate_to.lu opus_mt_lu_es
xx.fr.marian.translate_to.lus opus_mt_lus_fr
xx.tr.marian.translate_to.ja opus_mt_ja_tr
xx.fi.marian.translate_to.pag opus_mt_pag_fi
xx.fr.marian.translate_to.kqn opus_mt_kqn_fr
xx.fi.marian.translate_to.ja opus_mt_ja_fi
xx.af.marian.translate_to.nl opus_mt_nl_af
xx.sv.marian.translate_to.pag opus_mt_pag_sv
xx.sv.marian.translate_to.nl opus_mt_nl_sv
xx.uk.marian.translate_to.no opus_mt_no_uk
xx.es.marian.translate_to.lua opus_mt_lua_es
xx.fi.marian.translate_to.mt opus_mt_mt_fi
xx.eo.marian.translate_to.lt opus_mt_lt_eo
xx.de.marian.translate_to.no opus_mt_no_de
xx.eo.marian.translate_to.pl opus_mt_pl_eo
xx.es.marian.translate_to.loz opus_mt_loz_es
xx.ru.marian.translate_to.ja opus_mt_ja_ru
xx.sv.marian.translate_to.pl opus_mt_pl_sv
xx.fi.marian.translate_to.mh opus_mt_mh_fi
xx.hu.marian.translate_to.ja opus_mt_ja_hu
xx.fi.marian.translate_to.mk opus_mt_mk_fi
xx.es.marian.translate_to.lue opus_mt_lue_es
xx.sv.marian.translate_to.lt opus_mt_lt_sv
xx.fr.marian.translate_to.pon opus_mt_pon_fr
xx.es.marian.translate_to.pap opus_mt_pap_es
xx.es.marian.translate_to.ln opus_mt_ln_es
xx.de.marian.translate_to.loz opus_mt_loz_de
xx.ru.marian.translate_to.ka opus_mt_ka_ru
xx.sv.marian.translate_to.kwy opus_mt_kwy_sv
xx.fi.marian.translate_to.lv opus_mt_lv_fi
xx.pl.marian.translate_to.ja opus_mt_ja_pl
xx.hu.marian.translate_to.ko opus_mt_ko_hu
xx.de.marian.translate_to.ja opus_mt_ja_de
xx.de.marian.translate_to.ko opus_mt_ko_de
xx.es.marian.translate_to.kg opus_mt_kg_es
xx.de.marian.translate_to.pap opus_mt_pap_de
xx.fi.marian.translate_to.no opus_mt_no_fi
xx.fi.marian.translate_to.lue opus_mt_lue_fi
xx.no.marian.translate_to.pl opus_mt_pl_no
xx.fr.marian.translate_to.mt opus_mt_mt_fr
xx.es.marian.translate_to.mg opus_mt_mg_es
xx.es.marian.translate_to.pis opus_mt_pis_es
xx.fr.marian.translate_to.pl opus_mt_pl_fr
xx.sv.marian.translate_to.ko opus_mt_ko_sv
xx.sv.marian.translate_to.loz opus_mt_loz_sv
xx.fi.marian.translate_to.loz opus_mt_loz_fi
xx.pl.marian.translate_to.no opus_mt_no_pl
xx.nl.marian.translate_to.ja opus_mt_ja_nl
xx.de.marian.translate_to.pl opus_mt_pl_de
xx.lt.marian.translate_to.pl opus_mt_pl_lt
xx.ru.marian.translate_to.ko opus_mt_ko_ru
xx.fr.marian.translate_to.lv opus_mt_lv_fr
xx.he.marian.translate_to.ja opus_mt_ja_he
xx.sv.marian.translate_to.niu opus_mt_niu_sv
xx.de.marian.translate_to.ms opus_mt_ms_de
xx.es.marian.translate_to.lt opus_mt_lt_es
xx.sv.marian.translate_to.no opus_mt_no_sv
xx.nl.marian.translate_to.no opus_mt_no_nl
xx.fi.marian.translate_to.lua opus_mt_lua_fi
xx.fr.marian.translate_to.lt opus_mt_lt_fr
xx.ms.marian.translate_to.ja opus_mt_ja_ms
xx.es.marian.translate_to.kqn opus_mt_kqn_es
xx.fr.marian.translate_to.lg opus_mt_lg_fr
xx.es.marian.translate_to.mk opus_mt_mk_es
xx.da.marian.translate_to.no opus_mt_no_da
xx.it.marian.translate_to.lt opus_mt_lt_it
xx.es.marian.translate_to.prl opus_mt_prl_es
xx.fr.marian.translate_to.lua opus_mt_lua_fr
xx.es.marian.translate_to.nso opus_mt_nso_es
xx.sv.marian.translate_to.lv opus_mt_lv_sv
xx.fi.marian.translate_to.pis opus_mt_pis_fi
xx.es.marian.translate_to.pon opus_mt_pon_es
xx.fr.marian.translate_to.ko opus_mt_ko_fr
xx.de.marian.translate_to.ln opus_mt_ln_de
xx.uk.marian.translate_to.nl opus_mt_nl_uk
xx.eo.marian.translate_to.nl opus_mt_nl_eo
xx.es.marian.translate_to.lv opus_mt_lv_es
xx.tr.marian.translate_to.lt opus_mt_lt_tr
xx.es.marian.translate_to.mt opus_mt_mt_es
xx.fi.marian.translate_to.lus opus_mt_lus_fi
xx.tl.marian.translate_to.pt opus_mt_pt_tl
xx.no.marian.translate_to.nl opus_mt_nl_no
xx.sv.marian.translate_to.kqn opus_mt_kqn_sv
xx.pt.marian.translate_to.ja opus_mt_ja_pt
xx.fi.marian.translate_to.nso opus_mt_nso_fi
xx.fr.marian.translate_to.kg opus_mt_kg_fr
xx.sv.marian.translate_to.pis opus_mt_pis_sv
xx.is.marian.translate_to.sv opus_mt_sv_is
xx.sla.marian.translate_to.sla opus_mt_sla_sla
xx.sv.marian.translate_to.srn opus_mt_srn_sv
xx.niu.marian.translate_to.sv opus_mt_sv_niu
xx.to.marian.translate_to.sv opus_mt_sv_to
xx.guw.marian.translate_to.sv opus_mt_sv_guw
xx.sn.marian.translate_to.sv opus_mt_sv_sn
xx.sv.marian.translate_to.rnd opus_mt_rnd_sv
xx.tum.marian.translate_to.sv opus_mt_sv_tum
xx.mos.marian.translate_to.sv opus_mt_sv_mos
xx.srn.marian.translate_to.sv opus_mt_sv_srn
xx.ht.marian.translate_to.sv opus_mt_sv_ht
xx.no.marian.translate_to.ru opus_mt_ru_no
xx.sl.marian.translate_to.sv opus_mt_sv_sl
xx.fr.marian.translate_to.sv opus_mt_sv_fr
xx.uk.marian.translate_to.ru opus_mt_ru_uk
xx.tiv.marian.translate_to.sv opus_mt_sv_tiv
xx.es.marian.translate_to.ru opus_mt_ru_es
xx.pag.marian.translate_to.sv opus_mt_sv_pag
xx.gaa.marian.translate_to.sv opus_mt_sv_gaa
xx.kqn.marian.translate_to.sv opus_mt_sv_kqn
xx.fr.marian.translate_to.sg opus_mt_sg_fr
xx.st.marian.translate_to.sv opus_mt_sv_st
xx.ase.marian.translate_to.sv opus_mt_sv_ase
xx.es.marian.translate_to.rn opus_mt_rn_es
xx.ru.marian.translate_to.sl opus_mt_sl_ru
xx.lu.marian.translate_to.sv opus_mt_sv_lu
xx.eu.marian.translate_to.ru opus_mt_ru_eu
xx.no.marian.translate_to.sv opus_mt_sv_no
xx.sq.marian.translate_to.sv opus_mt_sv_sq
xx.da.marian.translate_to.ru opus_mt_ru_da
xx.ny.marian.translate_to.sv opus_mt_sv_ny
xx.kg.marian.translate_to.sv opus_mt_sv_kg
xx.pis.marian.translate_to.sv opus_mt_sv_pis
xx.sv.marian.translate_to.sk opus_mt_sk_sv
xx.lus.marian.translate_to.sv opus_mt_sv_lus
xx.fi.marian.translate_to.sl opus_mt_sl_fi
xx.tn.marian.translate_to.sv opus_mt_sv_tn
xx.fr.marian.translate_to.srn opus_mt_srn_fr
xx.lv.marian.translate_to.sv opus_mt_sv_lv
xx.uk.marian.translate_to.sl opus_mt_sl_uk
xx.sg.marian.translate_to.sv opus_mt_sv_sg
xx.he.marian.translate_to.sv opus_mt_sv_he
xx.eo.marian.translate_to.ru opus_mt_ru_eo
xx.fr.marian.translate_to.ru opus_mt_ru_fr
xx.lv.marian.translate_to.ru opus_mt_ru_lv
xx.lua.marian.translate_to.sv opus_mt_sv_lua
xx.ar.marian.translate_to.ru opus_mt_ru_ar
xx.tll.marian.translate_to.sv opus_mt_sv_tll
xx.lue.marian.translate_to.sv opus_mt_sv_lue
xx.bi.marian.translate_to.sv opus_mt_sv_bi
xx.hu.marian.translate_to.sv opus_mt_sv_hu
xx.bzs.marian.translate_to.sv opus_mt_sv_bzs
xx.ru.marian.translate_to.sv opus_mt_sv_ru
xx.eo.marian.translate_to.ro opus_mt_ro_eo
xx.es.marian.translate_to.st opus_mt_st_es
xx.mt.marian.translate_to.sv opus_mt_sv_mt
xx.af.marian.translate_to.sv opus_mt_sv_af
xx.ts.marian.translate_to.sv opus_mt_sv_ts
xx.af.marian.translate_to.ru opus_tatoeba_af_ru
xx.efi.marian.translate_to.sv opus_mt_sv_efi
xx.es.marian.translate_to.sv opus_mt_sv_es
xx.fi.marian.translate_to.sk opus_mt_sk_fi
xx.fr.marian.translate_to.rw opus_mt_rw_fr
xx.sv.marian.translate_to.run opus_mt_run_sv
xx.th.marian.translate_to.sv opus_mt_sv_th
xx.ln.marian.translate_to.sv opus_mt_sv_ln
xx.es.marian.translate_to.sk opus_mt_sk_es
xx.lt.marian.translate_to.ru opus_mt_ru_lt
xx.mfe.marian.translate_to.sv opus_mt_sv_mfe
xx.cs.marian.translate_to.sv opus_mt_sv_cs
xx.vi.marian.translate_to.ru opus_mt_ru_vi
xx.ee.marian.translate_to.sv opus_mt_sv_ee
xx.bg.marian.translate_to.ru opus_mt_ru_bg
xx.nso.marian.translate_to.sv opus_mt_sv_nso
xx.mh.marian.translate_to.sv opus_mt_sv_mh
xx.iso.marian.translate_to.sv opus_mt_sv_iso
xx.fi.marian.translate_to.st opus_mt_st_fi
xx.bg.marian.translate_to.sv opus_mt_sv_bg
xx.sv.marian.translate_to.sq opus_mt_sq_sv
xx.sv.marian.translate_to.sn opus_mt_sn_sv
xx.de.marian.translate_to.rn opus_mt_rn_de
xx.pon.marian.translate_to.sv opus_mt_sv_pon
xx.ha.marian.translate_to.sv opus_mt_sv_ha
xx.fi.marian.translate_to.ru opus_mt_ru_fi
xx.sk.marian.translate_to.sv opus_mt_sv_sk
xx.es.marian.translate_to.run opus_mt_run_es
xx.et.marian.translate_to.ru opus_mt_ru_et
xx.swc.marian.translate_to.sv opus_mt_sv_swc
xx.hil.marian.translate_to.sv opus_mt_sv_hil
xx.ro.marian.translate_to.sv opus_mt_sv_ro
xx.fr.marian.translate_to.rnd opus_mt_rnd_fr
xx.kwy.marian.translate_to.sv opus_mt_sv_kwy
xx.uk.marian.translate_to.sh opus_mt_sh_uk
xx.sm.marian.translate_to.sv opus_mt_sv_sm
xx.sv.marian.translate_to.rw opus_mt_rw_sv
xx.et.marian.translate_to.sv opus_mt_sv_et
xx.eo.marian.translate_to.sv opus_mt_sv_eo
xx.rnd.marian.translate_to.sv opus_mt_sv_rnd
xx.eo.marian.translate_to.sh opus_mt_sh_eo
xx.ru.marian.translate_to.rn opus_mt_rn_ru
xx.rw.marian.translate_to.sv opus_mt_sv_rw
xx.fr.marian.translate_to.sn opus_mt_sn_fr
xx.ig.marian.translate_to.sv opus_mt_sv_ig
xx.fj.marian.translate_to.sv opus_mt_sv_fj
xx.sl.marian.translate_to.ru opus_mt_ru_sl
xx.ho.marian.translate_to.sv opus_mt_sv_ho
xx.sv.marian.translate_to.sl opus_mt_sl_sv
xx.pap.marian.translate_to.sv opus_mt_sv_pap
xx.fr.marian.translate_to.sl opus_mt_sl_fr
xx.es.marian.translate_to.sl opus_mt_sl_es
xx.run.marian.translate_to.sv opus_mt_sv_run
xx.el.marian.translate_to.sv opus_mt_sv_el
xx.gil.marian.translate_to.sv opus_mt_sv_gil
xx.crs.marian.translate_to.sv opus_mt_sv_crs
xx.fr.marian.translate_to.sk opus_mt_sk_fr
xx.es.marian.translate_to.sq opus_mt_sq_es
xx.sv.marian.translate_to.sg opus_mt_sg_sv
xx.es.marian.translate_to.srn opus_mt_srn_es
xx.fr.marian.translate_to.ro opus_mt_ro_fr
xx.fr.marian.translate_to.rn opus_mt_rn_fr
xx.fr.marian.translate_to.st opus_mt_st_fr
xx.es.marian.translate_to.rw opus_mt_rw_es
xx.hr.marian.translate_to.sv opus_mt_sv_hr
xx.es.marian.translate_to.sm opus_mt_sm_es
xx.es.marian.translate_to.ssp opus_mt_ssp_es
xx.nl.marian.translate_to.sv opus_mt_sv_nl
xx.bem.marian.translate_to.sv opus_mt_sv_bem
xx.sem.marian.translate_to.sem opus_mt_sem_sem
xx.sv.marian.translate_to.sv opus_mt_sv_sv
xx.sv.marian.translate_to.st opus_mt_st_sv
xx.lg.marian.translate_to.sv opus_mt_sv_lg
xx.bcl.marian.translate_to.sv opus_mt_sv_bcl
xx.toi.marian.translate_to.sv opus_mt_sv_toi
xx.id.marian.translate_to.sv opus_mt_sv_id
xx.he.marian.translate_to.ru opus_mt_ru_he
xx.ceb.marian.translate_to.sv opus_mt_sv_ceb
xx.tw.marian.translate_to.sv opus_mt_sv_tw
xx.chk.marian.translate_to.sv opus_mt_sv_chk
xx.fr.marian.translate_to.sm opus_mt_sm_fr
xx.tvl.marian.translate_to.sv opus_mt_sv_tvl
xx.es.marian.translate_to.sg opus_mt_sg_es
xx.ilo.marian.translate_to.sv opus_mt_sv_ilo
xx.sv.marian.translate_to.ro opus_mt_ro_sv
xx.fi.marian.translate_to.sg opus_mt_sg_fi
xx.hy.marian.translate_to.ru opus_mt_ru_hy
xx.fi.marian.translate_to.ro opus_mt_ro_fi
xx.tpi.marian.translate_to.sv opus_mt_sv_tpi
xx.fi.marian.translate_to.sv opus_mt_sv_fi
xx.sv.marian.translate_to.ru opus_mt_ru_sv
xx.es.marian.translate_to.toi opus_mt_toi_es
xx.no.marian.translate_to.uk opus_mt_uk_no
xx.ar.marian.translate_to.tr opus_mt_tr_ar
xx.he.marian.translate_to.uk opus_mt_uk_he
xx.sv.marian.translate_to.tvl opus_mt_tvl_sv
xx.uk.marian.translate_to.sv opus_mt_sv_uk
xx.fr.marian.translate_to.tvl opus_mt_tvl_fr
xx.bg.marian.translate_to.uk opus_mt_uk_bg
xx.fi.marian.translate_to.toi opus_mt_toi_fi
xx.ca.marian.translate_to.uk opus_mt_uk_ca
xx.fr.marian.translate_to.uk opus_mt_uk_fr
xx.eo.marian.translate_to.tr opus_mt_tr_eo
xx.uk.marian.translate_to.tr opus_mt_tr_uk
xx.es.marian.translate_to.tl opus_mt_tl_es
xx.es.marian.translate_to.tr opus_mt_tr_es
xx.it.marian.translate_to.uk opus_mt_uk_it
xx.fi.marian.translate_to.uk opus_mt_uk_fi
xx.lt.marian.translate_to.tr opus_mt_tr_lt
xx.es.marian.translate_to.swc opus_mt_swc_es
xx.umb.marian.translate_to.sv opus_mt_sv_umb
xx.sv.marian.translate_to.tw opus_mt_tw_sv
xx.urj.marian.translate_to.urj opus_mt_urj_urj
xx.yap.marian.translate_to.sv opus_mt_sv_yap
xx.fr.marian.translate_to.ty opus_mt_ty_fr
xx.fr.marian.translate_to.swc opus_mt_swc_fr
xx.pt.marian.translate_to.tl opus_mt_tl_pt
xx.tr.marian.translate_to.uk opus_mt_uk_tr
xx.sv.marian.translate_to.tr opus_mt_tr_sv
xx.fi.marian.translate_to.tvl opus_mt_tvl_fi
xx.es.marian.translate_to.tn opus_mt_tn_es
xx.fi.marian.translate_to.swc opus_mt_swc_fi
xx.fr.marian.translate_to.toi opus_mt_toi_fr
xx.fi.marian.translate_to.ts opus_mt_ts_fi
xx.de.marian.translate_to.uk opus_mt_uk_de
xx.sv.marian.translate_to.uk opus_mt_uk_sv
xx.fi.marian.translate_to.tw opus_mt_tw_fi
xx.sv.marian.translate_to.to opus_mt_to_sv
xx.sv.marian.translate_to.tll opus_mt_tll_sv
xx.fr.marian.translate_to.th opus_mt_th_fr
xx.es.marian.translate_to.ty opus_mt_ty_es
xx.fr.marian.translate_to.tw opus_mt_tw_fr
xx.fr.marian.translate_to.to opus_mt_to_fr
xx.sl.marian.translate_to.uk opus_mt_uk_sl
xx.xh.marian.translate_to.sv opus_mt_sv_xh
xx.war.marian.translate_to.sv opus_mt_sv_war
xx.hu.marian.translate_to.uk opus_mt_uk_hu
xx.ru.marian.translate_to.uk opus_mt_uk_ru
xx.sv.marian.translate_to.tn opus_mt_tn_sv
xx.fr.marian.translate_to.tum opus_mt_tum_fr
xx.sv.marian.translate_to.toi opus_mt_toi_sv
xx.sv.marian.translate_to.ty opus_mt_ty_sv
xx.fr.marian.translate_to.tr opus_mt_tr_fr
xx.fr.marian.translate_to.tn opus_mt_tn_fr
xx.cs.marian.translate_to.uk opus_mt_uk_cs
xx.fr.marian.translate_to.ts opus_mt_ts_fr
xx.sv.marian.translate_to.swc opus_mt_swc_sv
xx.es.marian.translate_to.to opus_mt_to_es
xx.es.marian.translate_to.uk opus_mt_uk_es
xx.nl.marian.translate_to.uk opus_mt_uk_nl
xx.zne.marian.translate_to.sv opus_mt_sv_zne
xx.es.marian.translate_to.tvl opus_mt_tvl_es
xx.pt.marian.translate_to.uk opus_mt_uk_pt
xx.fr.marian.translate_to.tiv opus_mt_tiv_fr
xx.fr.marian.translate_to.tll opus_mt_tll_fr
xx.sh.marian.translate_to.uk opus_mt_uk_sh
xx.wls.marian.translate_to.sv opus_mt_sv_wls
xx.ve.marian.translate_to.sv opus_mt_sv_ve
xx.es.marian.translate_to.tum opus_mt_tum_es
xx.fi.marian.translate_to.tll opus_mt_tll_fi
xx.es.marian.translate_to.tw opus_mt_tw_es
xx.sv.marian.translate_to.tiv opus_mt_tiv_sv
xx.fi.marian.translate_to.ty opus_mt_ty_fi
xx.pl.marian.translate_to.uk opus_mt_uk_pl
xx.sv.marian.translate_to.tpi opus_mt_tpi_sv
xx.az.marian.translate_to.tr opus_mt_tr_az
xx.es.marian.translate_to.tll opus_mt_tll_es
xx.ty.marian.translate_to.sv opus_mt_sv_ty
xx.tzo.marian.translate_to.es opus_mt_es_tzo
xx.sv.marian.translate_to.crs opus_mt_crs_sv
xx.es.marian.translate_to.zai opus_mt_zai_es
xx.niu.marian.translate_to.de opus_mt_de_niu
xx.sv.marian.translate_to.nso opus_mt_nso_sv
xx.fr.marian.translate_to.bg opus_mt_bg_fr
xx.es.marian.translate_to.lus opus_mt_lus_es
xx.es.marian.translate_to.nl opus_mt_nl_es
xx.fr.marian.translate_to.yo opus_mt_yo_fr
xx.sv.marian.translate_to.ilo opus_mt_ilo_sv
xx.es.marian.translate_to.ts opus_mt_ts_es
xx.run.marian.translate_to.fr opus_mt_fr_run
xx.to.marian.translate_to.es opus_mt_es_to
xx.ceb.marian.translate_to.fi opus_mt_fi_ceb
xx.it.marian.translate_to.ja opus_mt_ja_it
xx.es.marian.translate_to.sn opus_mt_sn_es
xx.yo.marian.translate_to.sv opus_mt_sv_yo
xx.tr.marian.translate_to.az opus_mt_az_tr
xx.fr.marian.translate_to.no opus_mt_no_fr
xx.tn.marian.translate_to.fr opus_mt_fr_tn
xx.id.marian.translate_to.fr opus_mt_fr_id
xx.de.marian.translate_to.ca opus_mt_ca_de
xx.sv.marian.translate_to.tum opus_mt_tum_sv
xx.ru.marian.translate_to.da opus_mt_da_ru
xx.de.marian.translate_to.tl opus_mt_tl_de
xx.eo.marian.translate_to.fr opus_mt_fr_eo
xx.vi.marian.translate_to.zh opus_mt_zh_vi
xx.es.marian.translate_to.vi opus_mt_vi_es
xx.es.marian.translate_to.mfe opus_mt_mfe_es
xx.fi.marian.translate_to.iso opus_mt_iso_fi
xx.es.marian.translate_to.tzo opus_mt_tzo_es
xx.sn.marian.translate_to.es opus_mt_es_sn
xx.es.marian.translate_to.xh opus_mt_xh_es
xx.sv.marian.translate_to.zne opus_mt_zne_sv
xx.sv.marian.translate_to.ts opus_mt_ts_sv
xx.it.marian.translate_to.zh opus_mt_zh_it
xx.uk.marian.translate_to.zh opus_mt_zh_uk
xx.fi.marian.translate_to.yo opus_mt_yo_fi
xx.sv.marian.translate_to.war opus_mt_war_sv
xx.sv.marian.translate_to.yo opus_mt_yo_sv
xx.tll.marian.translate_to.es opus_mt_es_tll
xx.nl.marian.translate_to.zh opus_mt_zh_nl
xx.fr.marian.translate_to.wls opus_mt_wls_fr
xx.it.marian.translate_to.vi opus_mt_vi_it
xx.bg.marian.translate_to.zh opus_mt_zh_bg
xx.sv.marian.translate_to.xh opus_mt_xh_sv
xx.es.marian.translate_to.zne opus_mt_zne_es
xx.zlw.marian.translate_to.zlw opus_mt_zlw_zlw
xx.sv.marian.translate_to.yap opus_mt_yap_sv
xx.he.marian.translate_to.zh opus_mt_zh_he
xx.fr.marian.translate_to.xh opus_mt_xh_fr
xx.fi.marian.translate_to.war opus_mt_war_fi
xx.sv.marian.translate_to.zh opus_mt_zh_sv
xx.zls.marian.translate_to.zls opus_mt_zls_zls
xx.fi.marian.translate_to.zne opus_mt_zne_fi
xx.es.marian.translate_to.ve opus_mt_ve_es
xx.de.marian.translate_to.vi opus_mt_vi_de
xx.eo.marian.translate_to.vi opus_mt_vi_eo
xx.sv.marian.translate_to.wls opus_mt_wls_sv
xx.es.marian.translate_to.war opus_mt_war_es
xx.ru.marian.translate_to.vi opus_mt_vi_ru
xx.ms.marian.translate_to.zh opus_mt_zh_ms
xx.fr.marian.translate_to.zne opus_mt_zne_fr
xx.fr.marian.translate_to.yap opus_mt_yap_fr
xx.de.marian.translate_to.zh opus_mt_zh_de
xx.es.marian.translate_to.yo opus_mt_yo_es
xx.es.marian.translate_to.vsl opus_mt_vsl_es
xx.zle.marian.translate_to.zle opus_mt_zle_zle
xx.fr.marian.translate_to.vi opus_mt_vi_fr
xx.fr.marian.translate_to.war opus_mt_war_fr
xx.fi.marian.translate_to.zh opus_mt_zh_fi
xx.he.marian.translate_to.it opus_tatoeba_he_it
xx.es.marian.translate_to.zh opus_tatoeba_es_zh
xx.es.translate_to.af translate_af_es
xx.nl.translate_to.af translate_af_nl
xx.eo.translate_to.af translate_af_eo
xx.afa.translate_to.afa translate_afa_afa
xx.sv.translate_to.af translate_af_sv
xx.es.translate_to.aed translate_aed_es
xx.fr.translate_to.af translate_af_fr
xx.fi.translate_to.af translate_af_fi
xx.de.translate_to.af translate_af_de
xx.ru.translate_to.af translate_af_ru
xx.es.translate_to.az translate_az_es
xx.de.translate_to.bcl translate_bcl_de
xx.sv.translate_to.bem translate_bem_sv
xx.tr.translate_to.az translate_az_tr
xx.sv.translate_to.bcl translate_bcl_sv
xx.es.translate_to.ar translate_ar_es
xx.es.translate_to.bem translate_bem_es
xx.ru.translate_to.ar translate_ar_ru
xx.es.translate_to.be translate_be_es
xx.fr.translate_to.bem translate_bem_fr
xx.he.translate_to.ar translate_ar_he
xx.es.translate_to.bcl translate_bcl_es
xx.es.translate_to.ase translate_ase_es
xx.de.translate_to.ar translate_ar_de
xx.pl.translate_to.ar translate_ar_pl
xx.tr.translate_to.ar translate_ar_tr
xx.sv.translate_to.ase translate_ase_sv
xx.fi.translate_to.bcl translate_bcl_fi
xx.el.translate_to.ar translate_ar_el
xx.fr.translate_to.bcl translate_bcl_fr
xx.fi.translate_to.bem translate_bem_fi
xx.fr.translate_to.ase translate_ase_fr
xx.fr.translate_to.ar translate_ar_fr
xx.eo.translate_to.ar translate_ar_eo
xx.it.translate_to.ar translate_ar_it
xx.sv.translate_to.am translate_am_sv
xx.de.translate_to.ase translate_ase_de
xx.uk.translate_to.bg translate_bg_uk
xx.it.translate_to.bg translate_bg_it
xx.sv.translate_to.bzs translate_bzs_sv
xx.pt.translate_to.ca translate_ca_pt
xx.es.translate_to.ber translate_ber_es
xx.it.translate_to.ca translate_ca_it
xx.eo.translate_to.bg translate_bg_eo
xx.sv.translate_to.ceb translate_ceb_sv
xx.fr.translate_to.bi translate_bi_fr
xx.sv.translate_to.bg translate_bg_sv
xx.fr.translate_to.ca translate_ca_fr
xx.tr.translate_to.bg translate_bg_tr
xx.es.translate_to.ceb translate_ceb_es
xx.de.translate_to.ca translate_ca_de
xx.fi.translate_to.ceb translate_ceb_fi
xx.es.translate_to.ca translate_ca_es
xx.es.translate_to.bg translate_bg_es
xx.uk.translate_to.ca translate_ca_uk
xx.sv.translate_to.bi translate_bi_sv
xx.sv.translate_to.chk translate_chk_sv
xx.fr.translate_to.ceb translate_ceb_fr
xx.es.translate_to.bzs translate_bzs_es
xx.de.translate_to.crs translate_crs_de
xx.nl.translate_to.ca translate_ca_nl
xx.es.translate_to.chk translate_chk_es
xx.fr.translate_to.ber translate_ber_fr
xx.fi.translate_to.bzs translate_bzs_fi
xx.es.translate_to.crs translate_crs_es
xx.fi.translate_to.bg translate_bg_fi
xx.cpp.translate_to.cpp translate_cpp_cpp
xx.de.translate_to.bg translate_bg_de
xx.es.translate_to.bi translate_bi_es
xx.fr.translate_to.bzs translate_bzs_fr
xx.fr.translate_to.bg translate_bg_fr
xx.fr.translate_to.chk translate_chk_fr
xx.ru.translate_to.bg translate_bg_ru
xx.fi.translate_to.cs translate_cs_fi
xx.ha.translate_to.de translate_de_ha
xx.ee.translate_to.de translate_de_ee
xx.eo.translate_to.de translate_de_eo
xx.gil.translate_to.de translate_de_gil
xx.fj.translate_to.de translate_de_fj
xx.fr.translate_to.de translate_de_fr
xx.sv.translate_to.cs translate_cs_sv
xx.es.translate_to.csn translate_csn_es
xx.ru.translate_to.da translate_da_ru
xx.no.translate_to.da translate_da_no
xx.iso.translate_to.de translate_de_iso
xx.eu.translate_to.de translate_de_eu
xx.nl.translate_to.de translate_de_nl
xx.ilo.translate_to.de translate_de_ilo
xx.hr.translate_to.de translate_de_hr
xx.mt.translate_to.de translate_de_mt
xx.es.translate_to.da translate_da_es
xx.ar.translate_to.de translate_de_ar
xx.is.translate_to.de translate_de_is
xx.sv.translate_to.crs translate_crs_sv
xx.fr.translate_to.da translate_da_fr
xx.gaa.translate_to.de translate_de_gaa
xx.niu.translate_to.de translate_de_niu
xx.da.translate_to.de translate_de_da
xx.de.translate_to.da translate_da_de
xx.ase.translate_to.de translate_de_ase
xx.ig.translate_to.de translate_de_ig
xx.lua.translate_to.de translate_de_lua
xx.de.translate_to.de translate_de_de
xx.bi.translate_to.de translate_de_bi
xx.fr.translate_to.cs translate_cs_fr
xx.ms.translate_to.de translate_de_ms
xx.fi.translate_to.crs translate_crs_fi
xx.eo.translate_to.da translate_da_eo
xx.af.translate_to.de translate_de_af
xx.uk.translate_to.cs translate_cs_uk
xx.bg.translate_to.de translate_de_bg
xx.no.translate_to.de translate_de_no
xx.de.translate_to.cs translate_cs_de
xx.it.translate_to.de translate_de_it
xx.ho.translate_to.de translate_de_ho
xx.ln.translate_to.de translate_de_ln
xx.guw.translate_to.de translate_de_guw
xx.efi.translate_to.de translate_de_efi
xx.hil.translate_to.de translate_de_hil
xx.cs.translate_to.de translate_de_cs
xx.es.translate_to.csg translate_csg_es
xx.es.translate_to.de translate_de_es
xx.bcl.translate_to.de translate_de_bcl
xx.ht.translate_to.de translate_de_ht
xx.loz.translate_to.de translate_de_loz
xx.kg.translate_to.de translate_de_kg
xx.eo.translate_to.cs translate_cs_eo
xx.el.translate_to.de translate_de_el
xx.fi.translate_to.de translate_de_fi
xx.he.translate_to.de translate_de_he
xx.bzs.translate_to.de translate_de_bzs
xx.fr.translate_to.crs translate_crs_fr
xx.crs.translate_to.de translate_de_crs
xx.fi.translate_to.da translate_da_fi
xx.hu.translate_to.de translate_de_hu
xx.et.translate_to.de translate_de_et
xx.lt.translate_to.de translate_de_lt
xx.ca.translate_to.de translate_de_ca
xx.pl.translate_to.de translate_de_pl
xx.sv.translate_to.el translate_el_sv
xx.de.translate_to.ee translate_ee_de
xx.pag.translate_to.de translate_de_pag
xx.ar.translate_to.el translate_el_ar
xx.nso.translate_to.de translate_de_nso
xx.pon.translate_to.de translate_de_pon
xx.pap.translate_to.de translate_de_pap
xx.fr.translate_to.efi translate_efi_fr
xx.pis.translate_to.de translate_de_pis
xx.de.translate_to.efi translate_efi_de
xx.eo.translate_to.el translate_el_eo
xx.fi.translate_to.ee translate_ee_fi
xx.es.translate_to.ee translate_ee_es
xx.fr.translate_to.ee translate_ee_fr
xx.fi.translate_to.efi translate_efi_fi
xx.fr.translate_to.el translate_el_fr
xx.tl.translate_to.de translate_de_tl
xx.ny.translate_to.de translate_de_ny
xx.uk.translate_to.de translate_de_uk
xx.sv.translate_to.efi translate_efi_sv
xx.sv.translate_to.ee translate_ee_sv
xx.vi.translate_to.de translate_de_vi
xx.fi.translate_to.el translate_el_fi
xx.cs.translate_to.eo translate_eo_cs
xx.bzs.translate_to.es translate_es_bzs
xx.he.translate_to.eo translate_eo_he
xx.hu.translate_to.eo translate_eo_hu
xx.ro.translate_to.eo translate_eo_ro
xx.ber.translate_to.es translate_es_ber
xx.ca.translate_to.es translate_es_ca
xx.bcl.translate_to.es translate_es_bcl
xx.ceb.translate_to.es translate_es_ceb
xx.da.translate_to.eo translate_eo_da
xx.bi.translate_to.es translate_es_bi
xx.ee.translate_to.es translate_es_ee
xx.ru.translate_to.eo translate_eo_ru
xx.csg.translate_to.es translate_es_csg
xx.fi.translate_to.eo translate_eo_fi
xx.it.translate_to.eo translate_eo_it
xx.nl.translate_to.eo translate_eo_nl
xx.et.translate_to.es translate_es_et
xx.bg.translate_to.es translate_es_bg
xx.de.translate_to.eo translate_eo_de
xx.ar.translate_to.es translate_es_ar
xx.cs.translate_to.es translate_es_cs
xx.aed.translate_to.es translate_es_aed
xx.ase.translate_to.es translate_es_ase
xx.el.translate_to.es translate_es_el
xx.eo.translate_to.es translate_es_eo
xx.af.translate_to.eo translate_eo_af
xx.af.translate_to.es translate_es_af
xx.pl.translate_to.eo translate_eo_pl
xx.de.translate_to.es translate_es_de
xx.es.translate_to.eo translate_eo_es
xx.da.translate_to.es translate_es_da
xx.crs.translate_to.es translate_es_crs
xx.pt.translate_to.eo translate_eo_pt
xx.eu.translate_to.es translate_es_eu
xx.es.translate_to.es translate_es_es
xx.csn.translate_to.es translate_es_csn
xx.sv.translate_to.eo translate_eo_sv
xx.efi.translate_to.es translate_es_efi
xx.sh.translate_to.eo translate_eo_sh
xx.bg.translate_to.eo translate_eo_bg
xx.fr.translate_to.eo translate_eo_fr
xx.el.translate_to.eo translate_eo_el
xx.pl.translate_to.es translate_es_pl
xx.ro.translate_to.es translate_es_ro
xx.is.translate_to.es translate_es_is
xx.ln.translate_to.es translate_es_ln
xx.to.translate_to.es translate_es_to
xx.no.translate_to.es translate_es_no
xx.nl.translate_to.es translate_es_nl
xx.pag.translate_to.es translate_es_pag
xx.tvl.translate_to.es translate_es_tvl
xx.fr.translate_to.es translate_es_fr
xx.he.translate_to.es translate_es_he
xx.lus.translate_to.es translate_es_lus
xx.hil.translate_to.es translate_es_hil
xx.ny.translate_to.es translate_es_ny
xx.pap.translate_to.es translate_es_pap
xx.id.translate_to.es translate_es_id
xx.wls.translate_to.es translate_es_wls
xx.gaa.translate_to.es translate_es_gaa
xx.nso.translate_to.es translate_es_nso
xx.mk.translate_to.es translate_es_mk
xx.mt.translate_to.es translate_es_mt
xx.pis.translate_to.es translate_es_pis
xx.gl.translate_to.es translate_es_gl
xx.sn.translate_to.es translate_es_sn
xx.hr.translate_to.es translate_es_hr
xx.swc.translate_to.es translate_es_swc
xx.lua.translate_to.es translate_es_lua
xx.it.translate_to.es translate_es_it
xx.fj.translate_to.es translate_es_fj
xx.gil.translate_to.es translate_es_gil
xx.sm.translate_to.es translate_es_sm
xx.guw.translate_to.es translate_es_guw
xx.kg.translate_to.es translate_es_kg
xx.tl.translate_to.es translate_es_tl
xx.rn.translate_to.es translate_es_rn
xx.mfs.translate_to.es translate_es_mfs
xx.iso.translate_to.es translate_es_iso
xx.loz.translate_to.es translate_es_loz
xx.tpi.translate_to.es translate_es_tpi
xx.ha.translate_to.es translate_es_ha
xx.ht.translate_to.es translate_es_ht
xx.uk.translate_to.es translate_es_uk
xx.tw.translate_to.es translate_es_tw
xx.st.translate_to.es translate_es_st
xx.sg.translate_to.es translate_es_sg
xx.ilo.translate_to.es translate_es_ilo
xx.ru.translate_to.es translate_es_ru
xx.yo.translate_to.es translate_es_yo
xx.pon.translate_to.es translate_es_pon
xx.niu.translate_to.es translate_es_niu
xx.lt.translate_to.es translate_es_lt
xx.ty.translate_to.es translate_es_ty
xx.ig.translate_to.es translate_es_ig
xx.tzo.translate_to.es translate_es_tzo
xx.rw.translate_to.es translate_es_rw
xx.war.translate_to.es translate_es_war
xx.tll.translate_to.es translate_es_tll
xx.prl.translate_to.es translate_es_prl
xx.xh.translate_to.es translate_es_xh
xx.yua.translate_to.es translate_es_yua
xx.ho.translate_to.es translate_es_ho
xx.ve.translate_to.es translate_es_ve
xx.sl.translate_to.es translate_es_sl
xx.tn.translate_to.es translate_es_tn
xx.vi.translate_to.es translate_es_vi
xx.srn.translate_to.es translate_es_srn
xx.fi.translate_to.es translate_es_fi
xx.lua.translate_to.fi translate_fi_lua
xx.ny.translate_to.fi translate_fi_ny
xx.pon.translate_to.fi translate_fi_pon
xx.crs.translate_to.fi translate_fi_crs
xx.nso.translate_to.fi translate_fi_nso
xx.iso.translate_to.fi translate_fi_iso
xx.kqn.translate_to.fi translate_fi_kqn
xx.gaa.translate_to.fi translate_fi_gaa
xx.ru.translate_to.eu translate_eu_ru
xx.eo.translate_to.fi translate_fi_eo
xx.ig.translate_to.fi translate_fi_ig
xx.bem.translate_to.fi translate_fi_bem
xx.es.translate_to.et translate_et_es
xx.fj.translate_to.fi translate_fi_fj
xx.et.translate_to.fi translate_fi_et
xx.bcl.translate_to.fi translate_fi_bcl
xx.fi.translate_to.fi translate_fi_fi
xx.el.translate_to.fi translate_fi_el
xx.efi.translate_to.fi translate_fi_efi
xx.ht.translate_to.fi translate_fi_ht
xx.ceb.translate_to.fi translate_fi_ceb
xx.lg.translate_to.fi translate_fi_lg
xx.pap.translate_to.fi translate_fi_pap
xx.kg.translate_to.fi translate_fi_kg
xx.ee.translate_to.fi translate_fi_ee
xx.lv.translate_to.fi translate_fi_lv
xx.fr.translate_to.et translate_et_fr
xx.de.translate_to.et translate_et_de
xx.bzs.translate_to.fi translate_fi_bzs
xx.mos.translate_to.fi translate_fi_mos
xx.zh.translate_to.es translate_es_zh
xx.id.translate_to.fi translate_fi_id
xx.gil.translate_to.fi translate_fi_gil
xx.pis.translate_to.fi translate_fi_pis
xx.no.translate_to.fi translate_fi_no
xx.it.translate_to.fi translate_fi_it
xx.es.translate_to.fi translate_fi_es
xx.ha.translate_to.fi translate_fi_ha
xx.fr.translate_to.fi translate_fi_fr
xx.de.translate_to.fi translate_fi_de
xx.bg.translate_to.fi translate_fi_bg
xx.zai.translate_to.es translate_es_zai
xx.hil.translate_to.fi translate_fi_hil
xx.cs.translate_to.fi translate_fi_cs
xx.es.translate_to.eu translate_eu_es
xx.ilo.translate_to.fi translate_fi_ilo
xx.pag.translate_to.fi translate_fi_pag
xx.ln.translate_to.fi translate_fi_ln
xx.sv.translate_to.et translate_et_sv
xx.niu.translate_to.fi translate_fi_niu
xx.hr.translate_to.fi translate_fi_hr
xx.de.translate_to.eu translate_eu_de
xx.lus.translate_to.fi translate_fi_lus
xx.ru.translate_to.et translate_et_ru
xx.af.translate_to.fi translate_fi_af
xx.mh.translate_to.fi translate_fi_mh
xx.guw.translate_to.fi translate_fi_guw
xx.mfe.translate_to.fi translate_fi_mfe
xx.ho.translate_to.fi translate_fi_ho
xx.fse.translate_to.fi translate_fi_fse
xx.lu.translate_to.fi translate_fi_lu
xx.hu.translate_to.fi translate_fi_hu
xx.mk.translate_to.fi translate_fi_mk
xx.nl.translate_to.fi translate_fi_nl
xx.mg.translate_to.fi translate_fi_mg
xx.mt.translate_to.fi translate_fi_mt
xx.he.translate_to.fi translate_fi_he
xx.fi.translate_to.et translate_et_fi
xx.is.translate_to.fi translate_fi_is
xx.lue.translate_to.fi translate_fi_lue
xx.guw.translate_to.fr translate_fr_guw
xx.ber.translate_to.fr translate_fr_ber
xx.uk.translate_to.fi translate_fi_uk
xx.efi.translate_to.fr translate_fr_efi
xx.tr.translate_to.fi translate_fi_tr
xx.tn.translate_to.fi translate_fi_tn
xx.es.translate_to.fr translate_fr_es
xx.srn.translate_to.fi translate_fi_srn
xx.bcl.translate_to.fr translate_fr_bcl
xx.sl.translate_to.fi translate_fi_sl
xx.ht.translate_to.fr translate_fr_ht
xx.zne.translate_to.fi translate_fi_zne
xx.de.translate_to.fr translate_fr_de
xx.war.translate_to.fi translate_fi_war
xx.tpi.translate_to.fi translate_fi_tpi
xx.ca.translate_to.fr translate_fr_ca
xx.yap.translate_to.fi translate_fi_yap
xx.sn.translate_to.fi translate_fi_sn
xx.hr.translate_to.fr translate_fr_hr
xx.gil.translate_to.fr translate_fr_gil
xx.id.translate_to.fr translate_fr_id
xx.sv.translate_to.fi translate_fi_sv
xx.toi.translate_to.fi translate_fi_toi
xx.sk.translate_to.fi translate_fi_sk
xx.he.translate_to.fr translate_fr_he
xx.sq.translate_to.fi translate_fi_sq
xx.ve.translate_to.fi translate_fi_ve
xx.tw.translate_to.fi translate_fi_tw
xx.tvl.translate_to.fi translate_fi_tvl
xx.hil.translate_to.fr translate_fr_hil
xx.sw.translate_to.fi translate_fi_sw
xx.eo.translate_to.fr translate_fr_eo
xx.xh.translate_to.fi translate_fi_xh
xx.bi.translate_to.fr translate_fr_bi
xx.ru.translate_to.fi translate_fi_ru
xx.ceb.translate_to.fr translate_fr_ceb
xx.ig.translate_to.fr translate_fr_ig
xx.el.translate_to.fr translate_fr_el
xx.sm.translate_to.fi translate_fi_sm
xx.to.translate_to.fi translate_fi_to
xx.ase.translate_to.fr translate_fr_ase
xx.yo.translate_to.fi translate_fi_yo
xx.sg.translate_to.fi translate_fi_sg
xx.rw.translate_to.fi translate_fi_rw
xx.ts.translate_to.fi translate_fi_ts
xx.wls.translate_to.fi translate_fi_wls
xx.ho.translate_to.fr translate_fr_ho
xx.tll.translate_to.fi translate_fi_tll
xx.st.translate_to.fi translate_fi_st
xx.fiu.translate_to.fiu translate_fiu_fiu
xx.ro.translate_to.fi translate_fi_ro
xx.tiv.translate_to.fi translate_fi_tiv
xx.ha.translate_to.fr translate_fr_ha
xx.ee.translate_to.fr translate_fr_ee
xx.gaa.translate_to.fr translate_fr_gaa
xx.hu.translate_to.fr translate_fr_hu
xx.ty.translate_to.fi translate_fi_ty
xx.fr.translate_to.fj translate_fj_fr
xx.run.translate_to.fi translate_fi_run
xx.bem.translate_to.fr translate_fr_bem
xx.bzs.translate_to.fr translate_fr_bzs
xx.fj.translate_to.fr translate_fr_fj
xx.ar.translate_to.fr translate_fr_ar
xx.swc.translate_to.fi translate_fi_swc
xx.crs.translate_to.fr translate_fr_crs
xx.bg.translate_to.fr translate_fr_bg
xx.af.translate_to.fr translate_fr_af
xx.loz.translate_to.fr translate_fr_loz
xx.st.translate_to.fr translate_fr_st
xx.tn.translate_to.fr translate_fr_tn
xx.srn.translate_to.fr translate_fr_srn
xx.to.translate_to.fr translate_fr_to
xx.sk.translate_to.fr translate_fr_sk
xx.tum.translate_to.fr translate_fr_tum
xx.ts.translate_to.fr translate_fr_ts
xx.iso.translate_to.fr translate_fr_iso
xx.sv.translate_to.fr translate_fr_sv
xx.mt.translate_to.fr translate_fr_mt
xx.pap.translate_to.fr translate_fr_pap
xx.wls.translate_to.fr translate_fr_wls
xx.lua.translate_to.fr translate_fr_lua
xx.ro.translate_to.fr translate_fr_ro
xx.tll.translate_to.fr translate_fr_tll
xx.ilo.translate_to.fr translate_fr_ilo
xx.ve.translate_to.fr translate_fr_ve
xx.ny.translate_to.fr translate_fr_ny
xx.tpi.translate_to.fr translate_fr_tpi
xx.uk.translate_to.fr translate_fr_uk
xx.ln.translate_to.fr translate_fr_ln
xx.mfe.translate_to.fr translate_fr_mfe
xx.lue.translate_to.fr translate_fr_lue
xx.mos.translate_to.fr translate_fr_mos
xx.pon.translate_to.fr translate_fr_pon
xx.tvl.translate_to.fr translate_fr_tvl
xx.run.translate_to.fr translate_fr_run
xx.pag.translate_to.fr translate_fr_pag
xx.sg.translate_to.fr translate_fr_sg
xx.no.translate_to.fr translate_fr_no
xx.ty.translate_to.fr translate_fr_ty
xx.tl.translate_to.fr translate_fr_tl
xx.sl.translate_to.fr translate_fr_sl
xx.tiv.translate_to.fr translate_fr_tiv
xx.rw.translate_to.fr translate_fr_rw
xx.lus.translate_to.fr translate_fr_lus
xx.swc.translate_to.fr translate_fr_swc
xx.sm.translate_to.fr translate_fr_sm
xx.pl.translate_to.fr translate_fr_pl
xx.kg.translate_to.fr translate_fr_kg
xx.niu.translate_to.fr translate_fr_niu
xx.lg.translate_to.fr translate_fr_lg
xx.ms.translate_to.fr translate_fr_ms
xx.nso.translate_to.fr translate_fr_nso
xx.war.translate_to.fr translate_fr_war
xx.xh.translate_to.fr translate_fr_xh
xx.pis.translate_to.fr translate_fr_pis
xx.tw.translate_to.fr translate_fr_tw
xx.kwy.translate_to.fr translate_fr_kwy
xx.rnd.translate_to.fr translate_fr_rnd
xx.vi.translate_to.fr translate_fr_vi
xx.lu.translate_to.fr translate_fr_lu
xx.mh.translate_to.fr translate_fr_mh
xx.ru.translate_to.fr translate_fr_ru
xx.sn.translate_to.fr translate_fr_sn
xx.kqn.translate_to.fr translate_fr_kqn
xx.ar.translate_to.he translate_he_ar
xx.de.translate_to.he translate_he_de
xx.es.translate_to.gil translate_gil_es
xx.de.translate_to.gaa translate_gaa_de
xx.fr.translate_to.hu translate_hu_fr
xx.fr.translate_to.gil translate_gil_fr
xx.de.translate_to.guw translate_guw_de
xx.fr.translate_to.ht translate_ht_fr
xx.uk.translate_to.he translate_he_uk
xx.fi.translate_to.hu translate_hu_fi
xx.uk.translate_to.hu translate_hu_uk
xx.zne.translate_to.fr translate_fr_zne
xx.sv.translate_to.gaa translate_gaa_sv
xx.es.translate_to.guw translate_guw_es
xx.gmq.translate_to.gmq translate_gmq_gmq
xx.fi.translate_to.hil translate_hil_fi
xx.fi.translate_to.guw translate_guw_fi
xx.es.translate_to.he translate_he_es
xx.ur.translate_to.hi translate_hi_ur
xx.de.translate_to.hil translate_hil_de
xx.gmw.translate_to.gmw translate_gmw_gmw
xx.fi.translate_to.gaa translate_gaa_fi
xx.fi.translate_to.he translate_he_fi
xx.eo.translate_to.hu translate_hu_eo
xx.fi.translate_to.ht translate_ht_fi
xx.yo.translate_to.fr translate_fr_yo
xx.sv.translate_to.hr translate_hr_sv
xx.fr.translate_to.ha translate_ha_fr
xx.fi.translate_to.ha translate_ha_fi
xx.sv.translate_to.ha translate_ha_sv
xx.pt.translate_to.gl translate_gl_pt
xx.fr.translate_to.guw translate_guw_fr
xx.es.translate_to.ht translate_ht_es
xx.de.translate_to.hu translate_hu_de
xx.sv.translate_to.ht translate_ht_sv
xx.es.translate_to.hr translate_hr_es
xx.fr.translate_to.gaa translate_gaa_fr
xx.ru.translate_to.he translate_he_ru
xx.es.translate_to.gl translate_gl_es
xx.ru.translate_to.hy translate_hy_ru
xx.fi.translate_to.gil translate_gil_fi
xx.sv.translate_to.hu translate_hu_sv
xx.sv.translate_to.gil translate_gil_sv
xx.fi.translate_to.fse translate_fse_fi
xx.gem.translate_to.gem translate_gem_gem
xx.es.translate_to.ha translate_ha_es
xx.it.translate_to.he translate_he_it
xx.sv.translate_to.guw translate_guw_sv
xx.sv.translate_to.he translate_he_sv
xx.yap.translate_to.fr translate_fr_yap
xx.fr.translate_to.hr translate_hr_fr
xx.eo.translate_to.he translate_he_eo
xx.es.translate_to.gaa translate_gaa_es
xx.fi.translate_to.hr translate_hr_fi
xx.fr.translate_to.he translate_he_fr
xx.fi.translate_to.ilo translate_ilo_fi
xx.sv.translate_to.iso translate_iso_sv
xx.he.translate_to.ja translate_ja_he
xx.fi.translate_to.id translate_id_fi
xx.de.translate_to.ja translate_ja_de
xx.he.translate_to.it translate_it_he
xx.it.translate_to.ja translate_ja_it
xx.is.translate_to.it translate_it_is
xx.bg.translate_to.ja translate_ja_bg
xx.de.translate_to.ig translate_ig_de
xx.bg.translate_to.it translate_it_bg
xx.es.translate_to.id translate_id_es
xx.fr.translate_to.id translate_id_fr
xx.es.translate_to.ja translate_ja_es
xx.sv.translate_to.ja translate_ja_sv
xx.es.translate_to.iso translate_iso_es
xx.es.translate_to.ilo translate_ilo_es
xx.it.translate_to.is translate_is_it
xx.sv.translate_to.it translate_it_sv
xx.sv.translate_to.is translate_is_sv
xx.ru.translate_to.ja translate_ja_ru
xx.es.translate_to.kg translate_kg_es
xx.fi.translate_to.ig translate_ig_fi
xx.fr.translate_to.iso translate_iso_fr
xx.de.translate_to.ko translate_ko_de
xx.sv.translate_to.ilo translate_ilo_sv
xx.es.translate_to.is translate_is_es
xx.da.translate_to.ja translate_ja_da
xx.nl.translate_to.ja translate_ja_nl
xx.inc.translate_to.inc translate_inc_inc
xx.de.translate_to.is translate_is_de
xx.fr.translate_to.is translate_is_fr
xx.lt.translate_to.it translate_it_lt
xx.sv.translate_to.ig translate_ig_sv
xx.de.translate_to.ilo translate_ilo_de
xx.ar.translate_to.it translate_it_ar
xx.fr.translate_to.kg translate_kg_fr
xx.vi.translate_to.ja translate_ja_vi
xx.ru.translate_to.ka translate_ka_ru
xx.uk.translate_to.it translate_it_uk
xx.vi.translate_to.it translate_it_vi
xx.ms.translate_to.it translate_it_ms
xx.ar.translate_to.ja translate_ja_ar
xx.eo.translate_to.is translate_is_eo
xx.ca.translate_to.it translate_it_ca
xx.sh.translate_to.ja translate_ja_sh
xx.fi.translate_to.ja translate_ja_fi
xx.iir.translate_to.iir translate_iir_iir
xx.itc.translate_to.itc translate_itc_itc
xx.ms.translate_to.ja translate_ja_ms
xx.fr.translate_to.it translate_it_fr
xx.fr.translate_to.ja translate_ja_fr
xx.pt.translate_to.ja translate_ja_pt
xx.eo.translate_to.it translate_it_eo
xx.fi.translate_to.iso translate_iso_fi
xx.pl.translate_to.ja translate_ja_pl
xx.tr.translate_to.ja translate_ja_tr
xx.es.translate_to.ig translate_ig_es
xx.fr.translate_to.ig translate_ig_fr
xx.sv.translate_to.id translate_id_sv
xx.hu.translate_to.ja translate_ja_hu
xx.sv.translate_to.kg translate_kg_sv
xx.es.translate_to.it translate_it_es
xx.ine.translate_to.ine translate_ine_ine
xx.de.translate_to.it translate_it_de
xx.fi.translate_to.is translate_is_fi
xx.es.translate_to.mk translate_mk_es
xx.es.translate_to.lue translate_lue_es
xx.es.translate_to.lv translate_lv_es
xx.fi.translate_to.lue translate_lue_fi
xx.es.translate_to.ln translate_ln_es
xx.fr.translate_to.loz translate_loz_fr
xx.sv.translate_to.kwy translate_kwy_sv
xx.es.translate_to.lus translate_lus_es
xx.fr.translate_to.lv translate_lv_fr
xx.fr.translate_to.lu translate_lu_fr
xx.de.translate_to.lt translate_lt_de
xx.tr.translate_to.lt translate_lt_tr
xx.fr.translate_to.lus translate_lus_fr
xx.es.translate_to.mg translate_mg_es
xx.sv.translate_to.lua translate_lua_sv
xx.fr.translate_to.lg translate_lg_fr
xx.fr.translate_to.kwy translate_kwy_fr
xx.es.translate_to.lt translate_lt_es
xx.sv.translate_to.ko translate_ko_sv
xx.es.translate_to.kqn translate_kqn_es
xx.fr.translate_to.ko translate_ko_fr
xx.sv.translate_to.kqn translate_kqn_sv
xx.fi.translate_to.ko translate_ko_fi
xx.es.translate_to.mh translate_mh_es
xx.fr.translate_to.lua translate_lua_fr
xx.it.translate_to.lt translate_lt_it
xx.sv.translate_to.lt translate_lt_sv
xx.es.translate_to.lu translate_lu_es
xx.fi.translate_to.lua translate_lua_fi
xx.fr.translate_to.kqn translate_kqn_fr
xx.de.translate_to.loz translate_loz_de
xx.fr.translate_to.ms translate_ms_fr
xx.fr.translate_to.lt translate_lt_fr
xx.ru.translate_to.lv translate_lv_ru
xx.ms.translate_to.ms translate_ms_ms
xx.sv.translate_to.lus translate_lus_sv
xx.fr.translate_to.lue translate_lue_fr
xx.fi.translate_to.lu translate_lu_fi
xx.eo.translate_to.lt translate_lt_eo
xx.fi.translate_to.mk translate_mk_fi
xx.es.translate_to.ko translate_ko_es
xx.sv.translate_to.lue translate_lue_sv
xx.pl.translate_to.lt translate_lt_pl
xx.es.translate_to.mfe translate_mfe_es
xx.fi.translate_to.loz translate_loz_fi
xx.sv.translate_to.loz translate_loz_sv
xx.ru.translate_to.ko translate_ko_ru
xx.fi.translate_to.lg translate_lg_fi
xx.fi.translate_to.mh translate_mh_fi
xx.sv.translate_to.lv translate_lv_sv
xx.hu.translate_to.ko translate_ko_hu
xx.es.translate_to.lua translate_lua_es
xx.fi.translate_to.lv translate_lv_fi
xx.ru.translate_to.lt translate_lt_ru
xx.de.translate_to.ms translate_ms_de
xx.fi.translate_to.lus translate_lus_fi
xx.es.translate_to.lg translate_lg_es
xx.de.translate_to.ln translate_ln_de
xx.es.translate_to.mfs translate_mfs_es
xx.fr.translate_to.mk translate_mk_fr
xx.fr.translate_to.ln translate_ln_fr
xx.es.translate_to.loz translate_loz_es
xx.sv.translate_to.lu translate_lu_sv
xx.it.translate_to.ms translate_ms_it
xx.sv.translate_to.lg translate_lg_sv
xx.ar.translate_to.pl translate_pl_ar
xx.fr.translate_to.ro translate_ro_fr
xx.sv.translate_to.niu translate_niu_sv
xx.eo.translate_to.pl translate_pl_eo
xx.nl.translate_to.no translate_no_nl
xx.es.translate_to.no translate_no_es
xx.es.translate_to.pag translate_pag_es
xx.ru.translate_to.rn translate_rn_ru
xx.sv.translate_to.pag translate_pag_sv
xx.uk.translate_to.pt translate_pt_uk
xx.uk.translate_to.pl translate_pl_uk
xx.de.translate_to.pl translate_pl_de
xx.sv.translate_to.nl translate_nl_sv
xx.fr.translate_to.no translate_no_fr
xx.es.translate_to.niu translate_niu_es
xx.uk.translate_to.no translate_no_uk
xx.lt.translate_to.pl translate_pl_lt
xx.tl.translate_to.pt translate_pt_tl
xx.gl.translate_to.pt translate_pt_gl
xx.da.translate_to.ru translate_ru_da
xx.da.translate_to.no translate_no_da
xx.uk.translate_to.nl translate_nl_uk
xx.sv.translate_to.pon translate_pon_sv
xx.fr.translate_to.pis translate_pis_fr
xx.fr.translate_to.niu translate_niu_fr
xx.af.translate_to.nl translate_nl_af
xx.fi.translate_to.nso translate_nso_fi
xx.fi.translate_to.pon translate_pon_fi
xx.de.translate_to.pap translate_pap_de
xx.de.translate_to.rn translate_rn_de
xx.es.translate_to.pon translate_pon_es
xx.es.translate_to.pis translate_pis_es
xx.ca.translate_to.pt translate_pt_ca
xx.sv.translate_to.rnd translate_rnd_sv
xx.sv.translate_to.pl translate_pl_sv
xx.ru.translate_to.no translate_no_ru
xx.fi.translate_to.niu translate_niu_fi
xx.de.translate_to.pag translate_pag_de
xx.fr.translate_to.pl translate_pl_fr
xx.fi.translate_to.no translate_no_fi
xx.pl.translate_to.no translate_no_pl
xx.de.translate_to.nso translate_nso_de
xx.fr.translate_to.rn translate_rn_fr
xx.sv.translate_to.nso translate_nso_sv
xx.sv.translate_to.ro translate_ro_sv
xx.no.translate_to.pl translate_pl_no
xx.fr.translate_to.nl translate_nl_fr
xx.es.translate_to.nso translate_nso_es
xx.no.translate_to.nl translate_nl_no
xx.fi.translate_to.pis translate_pis_fi
xx.ca.translate_to.nl translate_nl_ca
xx.es.translate_to.nl translate_nl_es
xx.es.translate_to.ny translate_ny_es
xx.fr.translate_to.pap translate_pap_fr
xx.fi.translate_to.nl translate_nl_fi
xx.sv.translate_to.no translate_no_sv
xx.fr.translate_to.pon translate_pon_fr
xx.fr.translate_to.rnd translate_rnd_fr
xx.es.translate_to.pap translate_pap_es
xx.es.translate_to.prl translate_prl_es
xx.eo.translate_to.ro translate_ro_eo
xx.sv.translate_to.pis translate_pis_sv
xx.af.translate_to.ru translate_ru_af
xx.fr.translate_to.nso translate_nso_fr
xx.eo.translate_to.pt translate_pt_eo
xx.ar.translate_to.ru translate_ru_ar
xx.fr.translate_to.mt translate_mt_fr
xx.es.translate_to.rn translate_rn_es
xx.sv.translate_to.mt translate_mt_sv
xx.de.translate_to.niu translate_niu_de
xx.es.translate_to.mt translate_mt_es
xx.es.translate_to.pl translate_pl_es
xx.fi.translate_to.pag translate_pag_fi
xx.de.translate_to.no translate_no_de
xx.de.translate_to.ny translate_ny_de
xx.fi.translate_to.mt translate_mt_fi
xx.no.translate_to.no translate_no_no
xx.eo.translate_to.nl translate_nl_eo
xx.bg.translate_to.ru translate_ru_bg
xx.fi.translate_to.pap translate_pap_fi
xx.fi.translate_to.ro translate_ro_fi
xx.sv.translate_to.st translate_st_sv
xx.kg.translate_to.sv translate_sv_kg
xx.sv.translate_to.sq translate_sq_sv
xx.ee.translate_to.sv translate_sv_ee
xx.es.translate_to.srn translate_srn_es
xx.lv.translate_to.ru translate_ru_lv
xx.cs.translate_to.sv translate_sv_cs
xx.ha.translate_to.sv translate_sv_ha
xx.kqn.translate_to.sv translate_sv_kqn
xx.fr.translate_to.rw translate_rw_fr
xx.fr.translate_to.sn translate_sn_fr
xx.eu.translate_to.ru translate_ru_eu
xx.fi.translate_to.st translate_st_fi
xx.efi.translate_to.sv translate_sv_efi
xx.ho.translate_to.sv translate_sv_ho
xx.id.translate_to.sv translate_sv_id
xx.eo.translate_to.sv translate_sv_eo
xx.guw.translate_to.sv translate_sv_guw
xx.sv.translate_to.sk translate_sk_sv
xx.fr.translate_to.srn translate_srn_fr
xx.ceb.translate_to.sv translate_sv_ceb
xx.es.translate_to.sq translate_sq_es
xx.sv.translate_to.rw translate_rw_sv
xx.is.translate_to.sv translate_sv_is
xx.es.translate_to.sm translate_sm_es
xx.bcl.translate_to.sv translate_sv_bcl
xx.kwy.translate_to.sv translate_sv_kwy
xx.es.translate_to.run translate_run_es
xx.el.translate_to.sv translate_sv_el
xx.es.translate_to.sk translate_sk_es
xx.iso.translate_to.sv translate_sv_iso
xx.lu.translate_to.sv translate_sv_lu
xx.af.translate_to.sv translate_sv_af
xx.bg.translate_to.sv translate_sv_bg
xx.fr.translate_to.sm translate_sm_fr
xx.hr.translate_to.sv translate_sv_hr
xx.sv.translate_to.sn translate_sn_sv
xx.no.translate_to.ru translate_ru_no
xx.fr.translate_to.sg translate_sg_fr
xx.es.translate_to.sl translate_sl_es
xx.bzs.translate_to.sv translate_sv_bzs
xx.fr.translate_to.st translate_st_fr
xx.hu.translate_to.sv translate_sv_hu
xx.sv.translate_to.sg translate_sg_sv
xx.sem.translate_to.sem translate_sem_sem
xx.uk.translate_to.sh translate_sh_uk
xx.ln.translate_to.sv translate_sv_ln
xx.fi.translate_to.sk translate_sk_fi
xx.ht.translate_to.sv translate_sv_ht
xx.es.translate_to.st translate_st_es
xx.fr.translate_to.ru translate_ru_fr
xx.chk.translate_to.sv translate_sv_chk
xx.fr.translate_to.sk translate_sk_fr
xx.lg.translate_to.sv translate_sv_lg
xx.sv.translate_to.srn translate_srn_sv
xx.crs.translate_to.sv translate_sv_crs
xx.uk.translate_to.ru translate_ru_uk
xx.et.translate_to.ru translate_ru_et
xx.et.translate_to.sv translate_sv_et
xx.es.translate_to.rw translate_rw_es
xx.sla.translate_to.sla translate_sla_sla
xx.ru.translate_to.sl translate_sl_ru
xx.fj.translate_to.sv translate_sv_fj
xx.es.translate_to.sn translate_sn_es
xx.lua.translate_to.sv translate_sv_lua
xx.hil.translate_to.sv translate_sv_hil
xx.es.translate_to.ru translate_ru_es
xx.lue.translate_to.sv translate_sv_lue
xx.gaa.translate_to.sv translate_sv_gaa
xx.hy.translate_to.ru translate_ru_hy
xx.bem.translate_to.sv translate_sv_bem
xx.sv.translate_to.run translate_run_sv
xx.gil.translate_to.sv translate_sv_gil
xx.lus.translate_to.sv translate_sv_lus
xx.he.translate_to.ru translate_ru_he
xx.vi.translate_to.ru translate_ru_vi
xx.he.translate_to.sv translate_sv_he
xx.sv.translate_to.ru translate_ru_sv
xx.fi.translate_to.ru translate_ru_fi
xx.es.translate_to.sv translate_sv_es
xx.es.translate_to.sg translate_sg_es
xx.eo.translate_to.ru translate_ru_eo
xx.lv.translate_to.sv translate_sv_lv
xx.fi.translate_to.sg translate_sg_fi
xx.es.translate_to.ssp translate_ssp_es
xx.ilo.translate_to.sv translate_sv_ilo
xx.fi.translate_to.sv translate_sv_fi
xx.lt.translate_to.ru translate_ru_lt
xx.bi.translate_to.sv translate_sv_bi
xx.sv.translate_to.sl translate_sl_sv
xx.fr.translate_to.sv translate_sv_fr
xx.uk.translate_to.sl translate_sl_uk
xx.fi.translate_to.sl translate_sl_fi
xx.sl.translate_to.ru translate_ru_sl
xx.ig.translate_to.sv translate_sv_ig
xx.ase.translate_to.sv translate_sv_ase
xx.eo.translate_to.sh translate_sh_eo
xx.fr.translate_to.sl translate_sl_fr
xx.es.translate_to.tl translate_tl_es
xx.sv.translate_to.tw translate_tw_sv
xx.lt.translate_to.tr translate_tr_lt
xx.fi.translate_to.tll translate_tll_fi
xx.sn.translate_to.sv translate_sv_sn
xx.tn.translate_to.sv translate_sv_tn
xx.sv.translate_to.toi translate_toi_sv
xx.uk.translate_to.sv translate_sv_uk
xx.tiv.translate_to.sv translate_sv_tiv
xx.sk.translate_to.sv translate_sv_sk
xx.ty.translate_to.sv translate_sv_ty
xx.es.translate_to.toi translate_toi_es
xx.rw.translate_to.sv translate_sv_rw
xx.ny.translate_to.sv translate_sv_ny
xx.rnd.translate_to.sv translate_sv_rnd
xx.es.translate_to.tn translate_tn_es
xx.sv.translate_to.tn translate_tn_sv
xx.es.translate_to.tvl translate_tvl_es
xx.pon.translate_to.sv translate_sv_pon
xx.ve.translate_to.sv translate_sv_ve
xx.fr.translate_to.tvl translate_tvl_fr
xx.es.translate_to.tum translate_tum_es
xx.run.translate_to.sv translate_sv_run
xx.de.translate_to.tl translate_tl_de
xx.fi.translate_to.tw translate_tw_fi
xx.es.translate_to.ty translate_ty_es
xx.fr.translate_to.toi translate_toi_fr
xx.sv.translate_to.tll translate_tll_sv
xx.sg.translate_to.sv translate_sv_sg
xx.az.translate_to.tr translate_tr_az
xx.es.translate_to.ts translate_ts_es
xx.fr.translate_to.ts translate_ts_fr
xx.fr.translate_to.th translate_th_fr
xx.zne.translate_to.sv translate_sv_zne
xx.tw.translate_to.sv translate_sv_tw
xx.mh.translate_to.sv translate_sv_mh
xx.pag.translate_to.sv translate_sv_pag
xx.fr.translate_to.tum translate_tum_fr
xx.no.translate_to.sv translate_sv_no
xx.ts.translate_to.sv translate_sv_ts
xx.mt.translate_to.sv translate_sv_mt
xx.yo.translate_to.sv translate_sv_yo
xx.fr.translate_to.to translate_to_fr
xx.sv.translate_to.sv translate_sv_sv
xx.fi.translate_to.toi translate_toi_fi
xx.ro.translate_to.sv translate_sv_ro
xx.es.translate_to.tw translate_tw_es
xx.niu.translate_to.sv translate_sv_niu
xx.uk.translate_to.tr translate_tr_uk
xx.to.translate_to.sv translate_sv_to
xx.fi.translate_to.ts translate_ts_fi
xx.tll.translate_to.sv translate_sv_tll
xx.fr.translate_to.tll translate_tll_fr
xx.pt.translate_to.tl translate_tl_pt
xx.nso.translate_to.sv translate_sv_nso
xx.sq.translate_to.sv translate_sv_sq
xx.sv.translate_to.tpi translate_tpi_sv
xx.yap.translate_to.sv translate_sv_yap
xx.sv.translate_to.tr translate_tr_sv
xx.fr.translate_to.swc translate_swc_fr
xx.nl.translate_to.sv translate_sv_nl
xx.fi.translate_to.ty translate_ty_fi
xx.fr.translate_to.tr translate_tr_fr
xx.sv.translate_to.tum translate_tum_sv
xx.swc.translate_to.sv translate_sv_swc
xx.fi.translate_to.swc translate_swc_fi
xx.eo.translate_to.tr translate_tr_eo
xx.xh.translate_to.sv translate_sv_xh
xx.sv.translate_to.tvl translate_tvl_sv
xx.sl.translate_to.sv translate_sv_sl
xx.tum.translate_to.sv translate_sv_tum
xx.es.translate_to.to translate_to_es
xx.fr.translate_to.tn translate_tn_fr
xx.sv.translate_to.ty translate_ty_sv
xx.sv.translate_to.swc translate_swc_sv
xx.mos.translate_to.sv translate_sv_mos
xx.ar.translate_to.tr translate_tr_ar
xx.ru.translate_to.sv translate_sv_ru
xx.srn.translate_to.sv translate_sv_srn
xx.pis.translate_to.sv translate_sv_pis
xx.pap.translate_to.sv translate_sv_pap
xx.tvl.translate_to.sv translate_sv_tvl
xx.sv.translate_to.to translate_to_sv
xx.th.translate_to.sv translate_sv_th
xx.war.translate_to.sv translate_sv_war
xx.sv.translate_to.ts translate_ts_sv
xx.fr.translate_to.tw translate_tw_fr
xx.st.translate_to.sv translate_sv_st
xx.fr.translate_to.tiv translate_tiv_fr
xx.tpi.translate_to.sv translate_sv_tpi
xx.fi.translate_to.tvl translate_tvl_fi
xx.fr.translate_to.ty translate_ty_fr
xx.sm.translate_to.sv translate_sv_sm
xx.es.translate_to.swc translate_swc_es
xx.sv.translate_to.tiv translate_tiv_sv
xx.toi.translate_to.sv translate_sv_toi
xx.mfe.translate_to.sv translate_sv_mfe
xx.wls.translate_to.sv translate_sv_wls
xx.umb.translate_to.sv translate_sv_umb
xx.es.translate_to.tr translate_tr_es
xx.es.translate_to.tll translate_tll_es
xx.pt.translate_to.uk translate_uk_pt
xx.it.translate_to.zh translate_zh_it
xx.no.translate_to.uk translate_uk_no
xx.sh.translate_to.uk translate_uk_sh
xx.sv.translate_to.wls translate_wls_sv
xx.pl.translate_to.uk translate_uk_pl
xx.es.translate_to.yo translate_yo_es
xx.es.translate_to.war translate_war_es
xx.sv.translate_to.zh translate_zh_sv
xx.tr.translate_to.uk translate_uk_tr
xx.fi.translate_to.war translate_war_fi
xx.de.translate_to.zh translate_zh_de
xx.uk.translate_to.zh translate_zh_uk
xx.eo.translate_to.vi translate_vi_eo
xx.bg.translate_to.zh translate_zh_bg
xx.es.translate_to.zne translate_zne_es
xx.fr.translate_to.uk translate_uk_fr
xx.zls.translate_to.zls translate_zls_zls
xx.fr.translate_to.yo translate_yo_fr
xx.bg.translate_to.uk translate_uk_bg
xx.fr.translate_to.xh translate_xh_fr
xx.ca.translate_to.uk translate_uk_ca
xx.fi.translate_to.zh translate_zh_fi
xx.es.translate_to.zai translate_zai_es
xx.es.translate_to.uk translate_uk_es
xx.nl.translate_to.uk translate_uk_nl
xx.sv.translate_to.yap translate_yap_sv
xx.he.translate_to.uk translate_uk_he
xx.sl.translate_to.uk translate_uk_sl
xx.es.translate_to.ve translate_ve_es
xx.zlw.translate_to.zlw translate_zlw_zlw
xx.es.translate_to.tzo translate_tzo_es
xx.hu.translate_to.uk translate_uk_hu
xx.de.translate_to.vi translate_vi_de
xx.fi.translate_to.yo translate_yo_fi
xx.ru.translate_to.uk translate_uk_ru
xx.ms.translate_to.zh translate_zh_ms
xx.urj.translate_to.urj translate_urj_urj
xx.it.translate_to.uk translate_uk_it
xx.sv.translate_to.war translate_war_sv
xx.fr.translate_to.wls translate_wls_fr
xx.zle.translate_to.zle translate_zle_zle
xx.vi.translate_to.zh translate_zh_vi
xx.es.translate_to.vsl translate_vsl_es
xx.fi.translate_to.zne translate_zne_fi
xx.fi.translate_to.uk translate_uk_fi
xx.ru.translate_to.vi translate_vi_ru
xx.nl.translate_to.zh translate_zh_nl
xx.sv.translate_to.xh translate_xh_sv
xx.es.translate_to.xh translate_xh_es
xx.he.translate_to.zh translate_zh_he
xx.fr.translate_to.war translate_war_fr
xx.fr.translate_to.zne translate_zne_fr
xx.sv.translate_to.yo translate_yo_sv
xx.fr.translate_to.vi translate_vi_fr
xx.it.translate_to.vi translate_vi_it
xx.sv.translate_to.zne translate_zne_sv
xx.fr.translate_to.yap translate_yap_fr
xx.cs.translate_to.uk translate_uk_cs
xx.es.translate_to.vi translate_vi_es
xx.de.translate_to.uk translate_uk_de
xx.sv.translate_to.uk translate_uk_sv

Bugfixes

Install NLU in 1 line!

* Install NLU on Google Colab : !wget https://setup.johnsnowlabs.com/nlu/colab.sh  -O - | bash
* Install NLU on Kaggle       : !wget https://setup.johnsnowlabs.com/nlu/kaggle.sh  -O - | bash
* Install NLU via Pip         : ! pip install nlu pyspark==3.0.3

NLU Version 3.0.2

This release contains examples and tutorials on how to visualize the 1000+ state-of-the-art NLP models provided by NLU in just 1 line of code in streamlit. It includes simple 1-liners you can sprinkle into your Streamlit app to for features like Dependency Trees, Named Entities (NER), text classification results, semantic simmilarity, embedding visualizations via ELMO, BERT, ALBERT, XLNET and much more . Additionally, improvements for T5, various resolvers have been added and models Farsi, Hebrew, Korean, and Turkish

This is the ultimate NLP research tool. You can visualize and compare the results of hundreds of context aware deep learning embeddings and compare them with classical vanilla embeddings like Glove and can see with your own eyes how context is encoded by transformer models like BERT or XLNETand many more ! Besides that, you can also compare the results of the 200+ NER models John Snow Labs provides and see how peformances changes with varrying ebeddings, like Contextual, Static and Domain Specific Embeddings.

Install

For detailed instructions refer to the NLU install documentation here
You need Open JDK 8 installed and the following python packages

pip install nlu streamlit pyspark==3.0.1 sklearn plotly 

Problems? Connect with us on Slack!

Impatient and want some action?

Just run this Streamlit app, you can use it to generate python code for each NLU-Streamlit building block

streamlit run https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/examples/streamlit/01_dashboard.py

Quick Starter cheat sheet - All you need to know in 1 picture for NLU + Streamlit

For NLU models to load, see the NLU Namespace or the John Snow Labs Modelshub or go straight to the source. NLU Streamlit Cheatsheet

Examples

Just try out any of these. You can use the first example to generate python-code snippets which you can recycle as building blocks in your streamlit apps!

Example: 01_dashboard

streamlit run https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/examples/streamlit/01_dashboard.py

Example: 02_NER

streamlit run https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/examples/streamlit/02_NER.py

Example: 03_text_similarity_matrix

streamlit run https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/examples/streamlit/03_text_similarity_matrix.py

Example: 04_dependency_tree

streamlit run https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/examples/streamlit/04_dependency_tree.py

Example: 05_classifiers

streamlit run https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/examples/streamlit/05_classifiers.py

Example: 06_token_features

streamlit run https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/examples/streamlit/06_token_features.py

How to use NLU?

All you need to know about NLU is that there is the nlu.load() method which returns a NLUPipeline object which has a .predict() that works on most common data types in the pydata stack like Pandas dataframes .
Ontop of that, there are various visualization methods a NLUPipeline provides easily integrate in Streamlit as re-usable components. viz() method

Overview of NLU + Streamlit buildingblocks

Method Description
nlu.load('<Model>').predict(data) Load any of the 1000+ models by providing the model name any predict on most Pythontic data strucutres like Pandas, strings, arrays of strings and more
nlu.load('<Model>').viz_streamlit(data) Display full NLU exploration dashboard, that showcases every feature avaiable with dropdown selectors for 1000+ models
nlu.load('<Model>').viz_streamlit_similarity([string1, string2]) Display similarity matrix and scalar similarity for every word embedding loaded and 2 strings.
nlu.load('<Model>').viz_streamlit_ner(data) Visualize predicted NER tags from Named Entity Recognizer model
nlu.load('<Model>').viz_streamlit_dep_tree(data) Visualize Dependency Tree together with Part of Speech labels
nlu.load('<Model>').viz_streamlit_classes(data) Display all extracted class features and confidences for every classifier loaded in pipeline
nlu.load('<Model>').viz_streamlit_token(data) Display all detected token features and informations in Streamlit
nlu.load('<Model>').viz(data, write_to_streamlit=True) Display the raw visualization without any UI elements. See viz docs for more info. By default all aplicable nlu model references will be shown.
nlu.enable_streamlit_caching() Enable caching the nlu.load() call. Once enabled, the nlu.load() method will automatically cached. This is recommended to run first and for large peformance gans

Detailed visualizer information and API docs

function pipe.viz_streamlit

Display a highly configurable UI that showcases almost every feature available for Streamlit visualization with model selection dropdowns in your applications.
Ths includes :

  • Similarity Matrix & Scalars & Embedding Information for any of the 100+ Word Embedding Models
  • NER visualizations for any of the 200+ Named entity recognizers
  • Labled & Unlabled Dependency Trees visualizations with Part of Speech Tags for any of the 100+ Part of Speech Models
  • Token informations predicted by any of the 1000+ models
  • Classification results predicted by any of the 100+ models classification models
  • Pipeline Configuration & Model Information & Link to John Snow Labs Modelshub for all loaded pipelines
  • Auto generate Python code that can be copy pasted to re-create the individual Streamlit visualization blocks. NlLU takes the first model specified as nlu.load() for the first visualization run.
    Once the Streamlit app is running, additional models can easily be added via the UI.
    It is recommended to run this first, since you can generate Python code snippets to recreate individual Streamlit visualization blocks
nlu.load('ner').viz_streamlit(['I love NLU and Streamlit!','I hate buggy software'])

NLU Streamlit UI Overview

function parameters pipe.viz_streamlit

Argument Type Default Description
text Union [str, List[str], pd.DataFrame, pd.Series] 'NLU and Streamlit go together like peanutbutter and jelly' Default text for the Classification, Named Entitiy Recognizer, Token Information and Dependency Tree visualizations
similarity_texts Union[List[str],Tuple[str,str]] ('Donald Trump Likes to part', 'Angela Merkel likes to party') Default texts for the Text similarity visualization. Should contain exactly 2 strings which will be compared token embedding wise. For each embedding active, a token wise similarity matrix and a similarity scalar
model_selection List[str] [] List of nlu references to display in the model selector, see the NLU Namespace or the John Snow Labs Modelshub or go straight to the source for more info
title str 'NLU ❤️ Streamlit - Prototype your NLP startup in 0 lines of code🚀' Title of the Streamlit app
sub_title str 'Play with over 1000+ scalable enterprise NLP models' Sub title of the Streamlit app
visualizers List[str] ( "dependency_tree", "ner", "similarity", "token_information", 'classification') Define which visualizations should be displayed. By default all visualizations are displayed.
show_models_info bool True Show information for every model loaded in the bottom of the Streamlit app.
show_model_select bool True Show a model selection dropdowns that makes any of the 1000+ models avaiable in 1 click
show_viz_selection bool False Show a selector in the sidebar which lets you configure which visualizations are displayed.
show_logo bool True Show logo
display_infos bool False Display additonal information about ISO codes and the NLU namespace structure.
set_wide_layout_CSS bool True Whether to inject custom CSS or not.
key str "NLU_streamlit" Key for the Streamlit elements drawn
model_select_position str 'side' Whether to output the positions of predictions or not, see pipe.predict(positions=true) for more info
show_code_snippets bool False Display Python code snippets above visualizations that can be used to re-create the visualization
num_similarity_cols int 2 How many columns should for the layout in Streamlit when rendering the similarity matrixes.

function pipe.viz_streamlit_classes

Visualize the predicted classes and their confidences and additional metadata to streamlit. Aplicable with any of the 100+ classifiers

nlu.load('sentiment').viz_streamlit_classes(['I love NLU and Streamlit!','I love buggy software', 'Sign up now get a chance to win 1000$ !', 'I am afraid of Snakes','Unicorns have been sighted on Mars!','Where is the next bus stop?'])

text_class1

function parameters pipe.viz_streamlit_classes

Argument Type Default Description
text Union[str,list,pd.DataFrame, pd.Series, pyspark.sql.DataFrame ] 'I love NLU and Streamlit and sunny days!' Text to predict classes for. Will predict on each input of the iteratable or dataframe if type is not str.
output_level Optional[str] document Outputlevel of NLU pipeline, see pipe.predict() docsmore info
include_text_col bool True Whether to include a e text column in the output table or just the prediction data
title Optional[str] Text Classification Title of the Streamlit building block that will be visualized to screen
metadata bool False whether to output addition metadata or not, see pipe.predict(meta=true) docs for more info
positions bool False whether to output the positions of predictions or not, see pipe.predict(positions=true) for more info
set_wide_layout_CSS bool True Whether to inject custom CSS or not.
key str "NLU_streamlit" Key for the Streamlit elements drawn
model_select_position str 'side' Whether to output the positions of predictions or not, see pipe.predict(positions=true) for more info
generate_code_sample bool False Display Python code snippets above visualizations that can be used to re-create the visualization
show_model_select bool True Show a model selection dropdowns that makes any of the 1000+ models avaiable in 1 click
show_logo bool True Show logo
display_infos bool False Display additonal information about ISO codes and the NLU namespace structure.

function pipe.viz_streamlit_ner

Visualize the predicted classes and their confidences and additional metadata to Streamlit. Aplicable with any of the 250+ NER models.
You can filter which NER tags to highlight via the dropdown in the main window.

Basic usage

nlu.load('ner').viz_streamlit_ner('Donald Trump from America and Angela Merkel from Germany dont share many views')

NER visualization

Example for coloring

# Color all entities of class GPE black
nlu.load('ner').viz_streamlit_ner('Donald Trump from America and Angela Merkel from Germany dont share many views',colors={'PERSON':'#6e992e', 'GPE':'#000000'})

NER coloring

function parameters pipe.viz_streamlit_ner

Argument Type Default Description
text str 'Donald Trump from America and Anegela Merkel from Germany do not share many views' Text to predict classes for.
ner_tags Optional[List[str]] None Tags to display. By default all tags will be displayed
show_label_select bool True Whether to include the label selector
show_table bool True Whether show to predicted pandas table or not
title Optional[str] 'Named Entities' Title of the Streamlit building block that will be visualized to screen
sub_title Optional[str] '"Recognize various Named Entities (NER) in text entered and filter them. You can select from over 100 languages in the dropdown. On the left side.",' Sub-title of the Streamlit building block that will be visualized to screen
colors Dict[str,str] {} Dict with KEY=ENTITY_LABEL and VALUE=COLOR_AS_HEX_CODE,which will change color of highlighted entities.See custom color labels docs for more info.
set_wide_layout_CSS bool True Whether to inject custom CSS or not.
key str "NLU_streamlit" Key for the Streamlit elements drawn
generate_code_sample bool False Display Python code snippets above visualizations that can be used to re-create the visualization
show_model_select bool True Show a model selection dropdowns that makes any of the 1000+ models avaiable in 1 click
model_select_position str 'side' Whether to output the positions of predictions or not, see pipe.predict(positions=true) for more info
show_text_input bool True Show text input field to input text in
show_logo bool True Show logo
display_infos bool False Display additonal information about ISO codes and the NLU namespace structure.

function pipe.viz_streamlit_dep_tree

Visualize a typed dependency tree, the relations between tokens and part of speech tags predicted. Aplicable with any of the 100+ Part of Speech(POS) models and dep tree model

nlu.load('dep.typed').viz_streamlit_dep_tree('POS tags define a grammatical label for each token and the Dependency Tree classifies Relations between the tokens')

Dependency Tree

function parameters pipe.viz_streamlit_dep_tree

Argument Type Default Description
text str 'Billy likes to swim' Text to predict classes for.
title Optional[str] 'Dependency Parse Tree & Part-of-speech tags' Title of the Streamlit building block that will be visualized to screen
set_wide_layout_CSS bool True Whether to inject custom CSS or not.
key str "NLU_streamlit" Key for the Streamlit elements drawn
generate_code_sample bool False Display Python code snippets above visualizations that can be used to re-create the visualization
set_wide_layout_CSS bool True Whether to inject custom CSS or not.
key str "NLU_streamlit" Key for the Streamlit elements drawn
generate_code_sample bool False Display Python code snippets above visualizations that can be used to re-create the visualization
show_model_select bool True Show a model selection dropdowns that makes any of the 1000+ models avaiable in 1 click
model_select_position str 'side' Whether to output the positions of predictions or not, see pipe.predict(positions=true) for more info
show_logo bool True Show logo
display_infos bool False Display additonal information about ISO codes and the NLU namespace structure.

function pipe.viz_streamlit_token

Visualize predicted token and text features for every model loaded. You can use this with any of the 1000+ models and select them from the left dropdown.

nlu.load('stemm pos spell').viz_streamlit_token('I liek pentut buttr and jelly !')

text_class1

function parameters pipe.viz_streamlit_token

Argument Type Default Description
text str 'NLU and Streamlit are great!' Text to predict token information for.
title Optional[str] 'Named Entities' Title of the Streamlit building block that will be visualized to screen
show_feature_select bool True Whether to include the token feature selector
features Optional[List[str]] None Features to to display. By default all Features will be displayed
metadata bool False Whether to output addition metadata or not, see pipe.predict(meta=true) docs for more info
output_level Optional[str] 'token' Outputlevel of NLU pipeline, see pipe.predict() docsmore info
positions bool False Whether to output the positions of predictions or not, see pipe.predict(positions=true) for more info
set_wide_layout_CSS bool True Whether to inject custom CSS or not.
key str "NLU_streamlit" Key for the Streamlit elements drawn
generate_code_sample bool False Display Python code snippets above visualizations that can be used to re-create the visualization
show_model_select bool True Show a model selection dropdowns that makes any of the 1000+ models avaiable in 1 click
model_select_position str 'side' Whether to output the positions of predictions or not, see pipe.predict(positions=true) for more info
show_logo bool True Show logo
display_infos bool False Display additonal information about ISO codes and the NLU namespace structure.

function pipe.viz_streamlit_similarity

  • Displays a similarity matrix, where x-axis is every token in the first text and y-axis is every token in the second text.
  • Index i,j in the matrix describes the similarity of token-i to token-j based on the loaded embeddings and distance metrics, based on Sklearns Pariwise Metrics.. See this article for more elaboration on similarities
  • Displays a dropdown selectors from which various similarity metrics and over 100 embeddings can be selected. -There will be one similarity matrix per metric and embedding pair selected. num_plots = num_metric*num_embeddings Also displays embedding vector information. Applicable with any of the 100+ Word Embedding models
nlu.load('bert').viz_streamlit_word_similarity(['I love love loooove NLU! <3','I also love love looove  Streamlit! <3'])

text_class1

function parameters pipe.viz_streamlit_similarity

Argument Type Default Description
texts str 'Donald Trump from America and Anegela Merkel from Germany do not share many views.' Text to predict token information for.
title Optional[str] 'Named Entities' Title of the Streamlit building block that will be visualized to screen
similarity_matrix bool None Whether to display similarity matrix or not
show_algo_select bool True Whether to show dist algo select or not
show_table bool True Whether show to predicted pandas table or not
threshold float 0.5 Threshold for displaying result red on screen
set_wide_layout_CSS bool True Whether to inject custom CSS or not.
key str "NLU_streamlit" Key for the Streamlit elements drawn
generate_code_sample bool False Display Python code snippets above visualizations that can be used to re-create the visualization
show_model_select bool True Show a model selection dropdowns that makes any of the 1000+ models avaiable in 1 click
model_select_position str 'side' Whether to output the positions of predictions or not, see pipe.predict(positions=true) for more info
write_raw_pandas bool False Write the raw pandas similarity df to streamlit
display_embed_information bool True Show additional embedding information like dimension, nlu_reference, spark_nlp_reference, sotrage_reference, modelhub link and more.
dist_metrics List[str] ['cosine'] Which distance metrics to apply. If multiple are selected, there will be multiple plots for each embedding and metric. num_plots = num_metric*num_embeddings. Can use multiple at the same time, any of of cityblock,cosine,euclidean,l2,l1,manhattan,nan_euclidean. Provided via Sklearn metrics.pairwise package
num_cols int 2 How many columns should for the layout in streamlit when rendering the similarity matrixes.
display_scalar_similarities bool False Display scalar simmilarities in an additional field.
display_similarity_summary bool False Display summary of all similarities for all embeddings and metrics.
show_logo bool True Show logo
display_infos bool False Display additonal information about ISO codes and the NLU namespace structure.

## In addition have added some new features to our T5 Transformer annotator to help with longer and more accurate text generation, trained some new multi-lingual models and pipelines in Farsi, Hebrew, Korean, and Turkish.

T5 Model Improvements

  • Add 6 new features to T5Transformer for longer and better text generation
    • doSample: Whether or not to use sampling; use greedy decoding otherwise
    • temperature: The value used to module the next token probabilities
    • topK: The number of highest probability vocabulary tokens to keep for top-k-filtering
    • topP: If set to float < 1, only the most probable tokens with probabilities that add up to top_p or higher are kept for generation
    • repetitionPenalty: The parameter for repetition penalty. 1.0 means no penalty. See CTRL: A Conditional Transformer Language Model for Controllable Generation paper for more details
    • noRepeatNgramSize: If set to int > 0, all ngrams of that size can only occur once

New Open Source Model in NLU 3.0.2

New multilingual models and pipelines for Farsi, Hebrew, Korean, and Turkish

Model NLU Reference Spark NLP Reference Lang
ClassifierDLModel tr.classify.news classifierdl_bert_news tr
UniversalSentenceEncoder xx.use.multi tfhub_use_multi xx
UniversalSentenceEncoder xx.use.multi_lg tfhub_use_multi_lg xx
Pipeline NLU Reference Spark NLP Reference Lang
PretrainedPipeline fa.ner.dl recognize_entities_dl fa
PretrainedPipeline he.explain_document explain_document_lg he
PretrainedPipeline ko.explain_document explain_document_lg ko

New Healthcare Models in NLU 3.0.2

Five new resolver models:

  • en.resolve.umls: This model returns CUI (concept unique identifier) codes for Clinical Findings, Medical Devices, Anatomical Structures and Injuries & Poisoning terms.
  • en.resolve.umls.findings: This model returns CUI (concept unique identifier) codes for 200K concepts from clinical findings.
  • en.resolve.loinc: Map clinical NER entities to LOINC codes using sbiobert.
  • en.resolve.loinc.bluebert: Map clinical NER entities to LOINC codes using sbluebert.
  • en.resolve.HPO: This model returns Human Phenotype Ontology (HPO) codes for phenotypic abnormalities encountered in human diseases. It also returns associated codes from the following vocabularies for each HPO code:

Related NLU Notebook

Model NLU Reference Spark NLP Reference
Resolver en.resolve.umls sbiobertresolve_umls_major_concepts
Resolver en.resolve.umls.findings sbiobertresolve_umls_findings
Resolver en.resolve.loinc sbiobertresolve_loinc
Resolver en.resolve.loinc.biobert sbiobertresolve_loinc
Resolver en.resolve.loinc.bluebert sbluebertresolve_loinc
Resolver en.resolve.HPO sbiobertresolve_HPO

en.resolve.HPO

nlu.load('med_ner.jsl.wip.clinical en.resolve.HPO').viz("""These disorders include cancer, bipolar disorder, schizophrenia, autism, Cri-du-chat syndrome,
 myopia, cortical cataract-linked Alzheimer's disease, and infectious diseases""")

text_class1

en.resolve.loinc.bluebert

nlu.load('med_ner.jsl.wip.clinical en.resolve.loinc.bluebert').viz("""A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years prior to presentation and
subsequent type two diabetes mellitus (TSS2DM), one prior episode of HTG-induced pancreatitis three years prior to presentation, associated with an acute 
hepatitis, and obesity with a body mass index (BMI) of 33.5 kg/m2, presented with a one-week history of polyuria, polydipsia, poor appetite, and vomiting.""")

text_class1

en.resolve.umls.findings

nlu.load('med_ner.jsl.wip.clinical en.resolve.umls.findings').viz("""A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years prior to presentation and
subsequent type two diabetes mellitus (TSS2DM), one prior episode of HTG-induced pancreatitis three years prior to presentation, associated with an acute 
hepatitis, and obesity with a body mass index (BMI) of 33.5 kg/m2, presented with a one-week history of polyuria, polydipsia, poor appetite, and vomiting."""
)

text_class1

en.resolve.umls

nlu.load('med_ner.jsl.wip.clinical en.resolve.umls').viz("""A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years prior to presentation and
subsequent type two diabetes mellitus (TSS2DM), one prior episode of HTG-induced pancreatitis three years prior to presentation, associated with an acute 
hepatitis, and obesity with a body mass index (BMI) of 33.5 kg/m2, presented with a one-week history of polyuria, polydipsia, poor appetite, and vomiting.""")

text_class1

en.resolve.loinc

nlu.load('med_ner.jsl.wip.clinical en.resolve.loinc').predict("""A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years prior to presentation and
subsequent type two diabetes mellitus (TSS2DM), one prior episode of HTG-induced pancreatitis three years prior to presentation, associated with an acute 
hepatitis, and obesity with a body mass index (BMI) of 33.5 kg/m2, presented with a one-week history of polyuria, polydipsia, poor appetite, and vomiting.""")

text_class1

en.resolve.loinc.biobert

nlu.load('med_ner.jsl.wip.clinical en.resolve.loinc.biobert').predict("""A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years prior to presentation and
subsequent type two diabetes mellitus (TSS2DM), one prior episode of HTG-induced pancreatitis three years prior to presentation, associated with an acute 
hepatitis, and obesity with a body mass index (BMI) of 33.5 kg/m2, presented with a one-week history of polyuria, polydipsia, poor appetite, and vomiting.""")

text_class1

Install NLU in 1 line!

* Install NLU on Google Colab : !wget https://setup.johnsnowlabs.com/nlu/colab.sh  -O - | bash
* Install NLU on Kaggle       : !wget https://setup.johnsnowlabs.com/nlu/kaggle.sh  -O - | bash
* Install NLU via Pip         : ! pip install nlu pyspark==3.0.1

NLU Version 3.0.1

We are very excited to announce NLU 3.0.1 has been released! This is one of the most visually appealing releases, with the integration of the Spark-NLP-Display library and visualizations for dependency trees, entity resolution, entity assertion, relationship between entities and named entity recognition. In addition to this, the schema of how columns are named by NLU has been reworked and all 140+ tutorial notebooks have been updated to reflect the latest changes in NLU 3.0.0+ Finally, new multilingual models for Afrikaans, Welsh, Maltese, Tamil, andVietnamese are now available.

New Features and Enhancements

  • 1 line to visualization for NER, Dependency, Resolution, Assertion and Relation via Spark-NLP-Display integration
  • Improved column naming schema
  • Over 140 + NLU tutorial Notebooks updated and improved to reflect latest changes in NLU 3.0.0 +
  • New multilingual models for Afrikaans, Welsh, Maltese, Tamil, andVietnamese

Improved Column Name generation

  • NLU categorized each internal component now with boolean labels for name_deductable and always_name_deductable .
  • Before generating column names, NLU checks wether each component is of unique in the pipeline or not. If a component is not unique in the pipe and there are multiple components of same type, i.e. multiple NER models, NLU will deduct a base name for the final output columns from the NLU reference each NER model is pointing to.
  • If on the other hand, there is only one NER model in the pipeline, only the default ner column prefixed will be generated.
  • For some components, like embeddings and classifiers are now defined as always_name_deductable, for those NLU will always try to infer a meaningful base name for the output columns.
  • Newly trained component output columns will now be prefixed with trained_<type> , for types pos , ner, cLassifier, sentiment and multi_classifier

Enhanced offline mode

  • You can still load a model from a path as usual with nlu.load(path=model_path) and output columns will be suffixed with from_disk
  • You can now optionally also specify request parameter during load a model from HDD, it will be used to deduct more meaningful column name suffixes, instead of from_disk, i.e. by calling nlu.load(request ='en.embed_sentence.biobert.pubmed_pmc_base_cased', path=model_path)

NLU visualization

The latest NLU release integrated the beautiful Spark-NLP-Display package visualizations. You do not need to worry about installing it, when you try to visualize something, NLU will check if Spark-NLP-Display is installed, if it is missing it will be dynamically installed into your python executable environment, so you don’t need to worry about anything!

See the visualization tutorial notebook and visualization docs for more info.

Cheat Sheet visualization

NER visualization

Applicable to any of the 100+ NER models! See here for an overview

nlu.load('ner').viz("Donald Trump from America and Angela Merkel from Germany don't share many oppinions.")

NER visualization

Dependency tree visualization

Visualizes the structure of the labeled dependency tree and part of speech tags

nlu.load('dep.typed').viz("Billy went to the mall")

Dependency Tree visualization

#Bigger Example
nlu.load('dep.typed').viz("Donald Trump from America and Angela Merkel from Germany don't share many oppinions but they both love John Snow Labs software")

Dependency Tree visualization

Assertion status visualization

Visualizes asserted statuses and entities.
Applicable to any of the 10 + Assertion models! See here for an overview

nlu.load('med_ner.clinical assert').viz("The MRI scan showed no signs of cancer in the left lung")

Assert visualization

#bigger example
data ='This is the case of a very pleasant 46-year-old Caucasian female, seen in clinic on 12/11/07 during which time MRI of the left shoulder showed no evidence of rotator cuff tear. She did have a previous MRI of the cervical spine that did show an osteophyte on the left C6-C7 level. Based on this, negative MRI of the shoulder, the patient was recommended to have anterior cervical discectomy with anterior interbody fusion at C6-C7 level. Operation, expected outcome, risks, and benefits were discussed with her. Risks include, but not exclusive of bleeding and infection, bleeding could be soft tissue bleeding, which may compromise airway and may result in return to the operating room emergently for evacuation of said hematoma. There is also the possibility of bleeding into the epidural space, which can compress the spinal cord and result in weakness and numbness of all four extremities as well as impairment of bowel and bladder function. However, the patient may develop deeper-seated infection, which may require return to the operating room. Should the infection be in the area of the spinal instrumentation, this will cause a dilemma since there might be a need to remove the spinal instrumentation and/or allograft. There is also the possibility of potential injury to the esophageus, the trachea, and the carotid artery. There is also the risks of stroke on the right cerebral circulation should an undiagnosed plaque be propelled from the right carotid. She understood all of these risks and agreed to have the procedure performed.'
nlu.load('med_ner.clinical assert').viz(data)

Assert visualization

Relationship between entities visualization

Visualizes the extracted entities between relationship.
Applicable to any of the 20 + Relation Extractor models See here for an overview

nlu.load('med_ner.jsl.wip.clinical relation.temporal_events').viz('The patient developed cancer after a mercury poisoning in 1999 ')

Entity Relation visualization

# bigger example
data = 'This is the case of a very pleasant 46-year-old Caucasian female, seen in clinic on 12/11/07 during which time MRI of the left shoulder showed no evidence of rotator cuff tear. She did have a previous MRI of the cervical spine that did show an osteophyte on the left C6-C7 level. Based on this, negative MRI of the shoulder, the patient was recommended to have anterior cervical discectomy with anterior interbody fusion at C6-C7 level. Operation, expected outcome, risks, and benefits were discussed with her. Risks include, but not exclusive of bleeding and infection, bleeding could be soft tissue bleeding, which may compromise airway and may result in return to the operating room emergently for evacuation of said hematoma. There is also the possibility of bleeding into the epidural space, which can compress the spinal cord and result in weakness and numbness of all four extremities as well as impairment of bowel and bladder function. However, the patient may develop deeper-seated infection, which may require return to the operating room. Should the infection be in the area of the spinal instrumentation, this will cause a dilemma since there might be a need to remove the spinal instrumentation and/or allograft. There is also the possibility of potential injury to the esophageus, the trachea, and the carotid artery. There is also the risks of stroke on the right cerebral circulation should an undiagnosed plaque be propelled from the right carotid. She understood all of these risks and agreed to have the procedure performed'
pipe = nlu.load('med_ner.jsl.wip.clinical relation.clinical').viz(data)

Entity Relation visualization

Entity Resolution visualization for chunks

Visualizes resolutions of entities Applicable to any of the 100+ Resolver models See here for an overview

nlu.load('med_ner.jsl.wip.clinical resolve_chunk.rxnorm.in').viz("He took Prevacid 30 mg  daily")

Chunk Resolution visualization

# bigger example
data = "This is an 82 - year-old male with a history of prior tobacco use , hypertension , chronic renal insufficiency , COPD , gastritis , and TIA who initially presented to Braintree with a non-ST elevation MI and Guaiac positive stools , transferred to St . Margaret\'s Center for Women & Infants for cardiac catheterization with PTCA to mid LAD lesion complicated by hypotension and bradycardia requiring Atropine , IV fluids and transient dopamine possibly secondary to vagal reaction , subsequently transferred to CCU for close monitoring , hemodynamically stable at the time of admission to the CCU ."
nlu.load('med_ner.jsl.wip.clinical resolve_chunk.rxnorm.in').viz(data)

Chunk Resolution visualization

Entity Resolution visualization for sentences

Visualizes resolutions of entities in sentences Applicable to any of the 100+ Resolver models See here for an overview

nlu.load('med_ner.jsl.wip.clinical resolve.icd10cm').viz('She was diagnosed with a respiratory congestion')

Sentence Resolution visualization

# bigger example
data = 'The patient is a 5-month-old infant who presented initially on Monday with a cold, cough, and runny nose for 2 days. Mom states she had no fever. Her appetite was good but she was spitting up a lot. She had no difficulty breathing and her cough was described as dry and hacky. At that time, physical exam showed a right TM, which was red. Left TM was okay. She was fairly congested but looked happy and playful. She was started on Amoxil and Aldex and we told to recheck in 2 weeks to recheck her ear. Mom returned to clinic again today because she got much worse overnight. She was having difficulty breathing. She was much more congested and her appetite had decreased significantly today. She also spiked a temperature yesterday of 102.6 and always having trouble sleeping secondary to congestion'
nlu.load('med_ner.jsl.wip.clinical resolve.icd10cm').viz(data)

Sentence Resolution visualization

Configure visualizations

Define custom colors for labels

Some entity and relation labels will be highlighted with a pre-defined color, which you can find here.
For labels that have no color defined, a random color will be generated.
You can define colors for labels manually, by specifying via the viz_colors parameter and defining hex color codes in a dictionary that maps labels to colors .

data = 'Dr. John Snow suggested that Fritz takes 5mg penicilin for his cough'
# Define custom colors for labels
viz_colors={'STRENGTH':'#800080', 'DRUG_BRANDNAME':'#77b5fe', 'GENDER':'#77ffe'}
nlu.load('med_ner.jsl.wip.clinical').viz(data,viz_colors =viz_colors)

define colors labels

Filter entities that get highlighted

By default every entity class will be visualized.
The labels_to_viz can be used to define a set of labels to highlight.
Applicable for ner, resolution and assert.

data = 'Dr. John Snow suggested that Fritz takes 5mg penicilin for his cough'
# Filter wich NER label to viz
labels_to_viz=['SYMPTOM']
nlu.load('med_ner.jsl.wip.clinical').viz(data,labels_to_viz=labels_to_viz)

filter labels

New models

New multilingual models for Afrikaans, Welsh, Maltese, Tamil, andVietnamese

nlu.load() Refrence Spark NLP Refrence
vi.lemma lemma
mt.lemma lemma
ta.lemma lemma
af.lemma lemma
af.pos pos_afribooms
cy.lemma lemma

Reworked and updated NLU tutorial notebooks

All of the 140+ NLU tutorial Notebooks have been updated and reworked to reflect the latest changes in NLU 3.0.0+

Bugfixes

  • Fixed a bug that caused resolution algorithms output level to be inferred incorrectly
  • Fixed a bug that caused stranger cols got dropped
  • Fixed a bug that caused endings to miss when .predict(position=True) was specified
  • Fixed a bug that caused pd.Series to be converted incorrectly internally
  • Fixed a bug that caused output level transformations to crash
  • Fixed a bug that caused verbose mode not to turn of properly after turning it on.
  • fixed a bug that caused some models to crash when loaded for HDD

  • 140+ updates tutorials
  • Updated visualization docs
  • Models Hub with new models
  • Spark NLP publications
  • NLU in Action
  • NLU documentation
  • Discussions Engage with other community members, share ideas, and show off how you use Spark NLP and NLU!

Install NLU in 1 line!aaa

* Install NLU on Google Colab : ! wget https://setup.johnsnowlabs.com/nlu/colab.sh  -O - | bash
* Install NLU on Kaggle       : ! wget https://setup.johnsnowlabs.com/nlu/kaggle.sh  -O - | bash
* Install NLU via Pip         : ! pip install nlu pyspark==3.0.3

200+ State of the Art Medical Models for NER, Entity Resolution, Relation Extraction, Assertion, Spark 3 and Python 3.8 support in NLU 3.0 Release and much more

We are incredible excited to announce the release of NLU 3.0.0 which makes most of John Snow Labs medical healthcare model available in just 1 line of code in NLU. These models are the most accurate in their domains and highly scalable in Spark clusters.
In addition, Spark 3.0.X and Spark 3.1.X is now supported, together with Python3.8

This is enabled by the the amazing Spark NLP3.0.1 and Spark NLP for Healthcare 3.0.1 releases.

New Features

  • Over 200 new models for the healthcare domain
  • 6 new classes of models, Assertion, Sentence/Chunk Resolvers, Relation Extractors, Medical NER models, De-Identificator Models
  • Spark 3.0.X and 3.1.X support
  • Python 3.8 Support
  • New Output level relation
  • 1 Line to install NLU just run !wget https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/scripts/colab_setup.sh -O - | bash
  • Various new EMR and Databricks versions supported
  • GPU Mode, more then 600% speedup by enabling GPU mode.
  • Authorized mode for licensed features

New Documentation

New Notebooks

AssertionDLModels

Language nlu.load() reference Spark NLP Model reference
English assert assertion_dl
English assert.biobert assertion_dl_biobert
English assert.healthcare assertion_dl_healthcare
English assert.large assertion_dl_large

New Word Embeddings

Language nlu.load() reference Spark NLP Model reference
English embed.glove.clinical embeddings_clinical
English embed.glove.biovec embeddings_biovec
English embed.glove.healthcare embeddings_healthcare
English embed.glove.healthcare_100d embeddings_healthcare_100d
English en.embed.glove.icdoem embeddings_icdoem
English en.embed.glove.icdoem_2ng embeddings_icdoem_2ng

Sentence Entity resolvers

Language nlu.load() reference Spark NLP Model reference
English embed_sentence.biobert.mli sbiobert_base_cased_mli
English resolve sbiobertresolve_cpt
English resolve.cpt sbiobertresolve_cpt
English resolve.cpt.augmented sbiobertresolve_cpt_augmented
English resolve.cpt.procedures_augmented sbiobertresolve_cpt_procedures_augmented
English resolve.hcc.augmented sbiobertresolve_hcc_augmented
English resolve.icd10cm sbiobertresolve_icd10cm
English resolve.icd10cm.augmented sbiobertresolve_icd10cm_augmented
English resolve.icd10cm.augmented_billable sbiobertresolve_icd10cm_augmented_billable_hcc
English resolve.icd10pcs sbiobertresolve_icd10pcs
English resolve.icdo sbiobertresolve_icdo
English resolve.rxcui sbiobertresolve_rxcui
English resolve.rxnorm sbiobertresolve_rxnorm
English resolve.snomed sbiobertresolve_snomed_auxConcepts
English resolve.snomed.aux_concepts sbiobertresolve_snomed_auxConcepts
English resolve.snomed.aux_concepts_int sbiobertresolve_snomed_auxConcepts_int
English resolve.snomed.findings sbiobertresolve_snomed_findings
English resolve.snomed.findings_int sbiobertresolve_snomed_findings_int

RelationExtractionModel

Language nlu.load() reference Spark NLP Model reference
English relation.posology posology_re
English relation redl_bodypart_direction_biobert
English relation.bodypart.direction redl_bodypart_direction_biobert
English relation.bodypart.problem redl_bodypart_problem_biobert
English relation.bodypart.procedure redl_bodypart_procedure_test_biobert
English relation.chemprot redl_chemprot_biobert
English relation.clinical redl_clinical_biobert
English relation.date redl_date_clinical_biobert
English relation.drug_drug_interaction redl_drug_drug_interaction_biobert
English relation.humen_phenotype_gene redl_human_phenotype_gene_biobert
English relation.temporal_events redl_temporal_events_biobert

NERDLModels

Language nlu.load() reference Spark NLP Model reference
English med_ner.ade.clinical ner_ade_clinical
English med_ner.ade.clinical_bert ner_ade_clinicalbert
English med_ner.ade.ade_healthcare ner_ade_healthcare
English med_ner.anatomy ner_anatomy
English med_ner.anatomy.biobert ner_anatomy_biobert
English med_ner.anatomy.coarse ner_anatomy_coarse
English med_ner.anatomy.coarse_biobert ner_anatomy_coarse_biobert
English med_ner.aspect_sentiment ner_aspect_based_sentiment
English med_ner.bacterial_species ner_bacterial_species
English med_ner.bionlp ner_bionlp
English med_ner.bionlp.biobert ner_bionlp_biobert
English med_ner.cancer ner_cancer_genetics
Englishs med_ner.cellular ner_cellular
English med_ner.cellular.biobert ner_cellular_biobert
English med_ner.chemicals ner_chemicals
English med_ner.chemprot ner_chemprot_biobert
English med_ner.chemprot.clinical ner_chemprot_clinical
English med_ner.clinical ner_clinical
English med_ner.clinical.biobert ner_clinical_biobert
English med_ner.clinical.noncontrib ner_clinical_noncontrib
English med_ner.diseases ner_diseases
English med_ner.diseases.biobert ner_diseases_biobert
English med_ner.diseases.large ner_diseases_large
English med_ner.drugs ner_drugs
English med_ner.drugsgreedy ner_drugs_greedy
English med_ner.drugs.large ner_drugs_large
English med_ner.events_biobert ner_events_biobert
English med_ner.events_clinical ner_events_clinical
English med_ner.events_healthcre ner_events_healthcare
English med_ner.financial_contract ner_financial_contract
English med_ner.healthcare ner_healthcare
English med_ner.human_phenotype.gene_biobert ner_human_phenotype_gene_biobert
English med_ner.human_phenotype.gene_clinical ner_human_phenotype_gene_clinical
English med_ner.human_phenotype.go_biobert ner_human_phenotype_go_biobert
English med_ner.human_phenotype.go_clinical ner_human_phenotype_go_clinical
English med_ner.jsl ner_jsl
English med_ner.jsl.biobert ner_jsl_biobert
English med_ner.jsl.enriched ner_jsl_enriched
English med_ner.jsl.enriched_biobert ner_jsl_enriched_biobert
English med_ner.measurements ner_measurements_clinical
English med_ner.medmentions ner_medmentions_coarse
English med_ner.posology ner_posology
English med_ner.posology.biobert ner_posology_biobert
English med_ner.posology.greedy ner_posology_greedy
English med_ner.posology.healthcare ner_posology_healthcare
English med_ner.posology.large ner_posology_large
English med_ner.posology.large_biobert ner_posology_large_biobert
English med_ner.posology.small ner_posology_small
English med_ner.radiology ner_radiology
English med_ner.radiology.wip_clinical ner_radiology_wip_clinical
English med_ner.risk_factors ner_risk_factors
English med_ner.risk_factors.biobert ner_risk_factors_biobert
English med_ner.i2b2 nerdl_i2b2
English med_ner.tumour nerdl_tumour_demo
English med_ner.jsl.wip.clinical jsl_ner_wip_clinical
English med_ner.jsl.wip.clinical.greedy jsl_ner_wip_greedy_clinical
English med_ner.jsl.wip.clinical.modifier jsl_ner_wip_modifier_clinical
English med_ner.jsl.wip.clinical.rd jsl_rd_ner_wip_greedy_clinical

De-Identification Models

Language nlu.load() reference Spark NLP Model reference
English med_ner.deid.augmented ner_deid_augmented
English med_ner.deid.biobert ner_deid_biobert
English med_ner.deid.enriched ner_deid_enriched
English med_ner.deid.enriched_biobert ner_deid_enriched_biobert
English med_ner.deid.large ner_deid_large
English med_ner.deid.sd ner_deid_sd
English med_ner.deid.sd_large ner_deid_sd_large
English med_ner.deid nerdl_deid
English med_ner.deid.synthetic ner_deid_synthetic
English med_ner.deid.dl ner_deidentify_dl
English en.de_identify deidentify_rb
English de_identify.rules deid_rules
English de_identify.clinical deidentify_enriched_clinical
English de_identify.large deidentify_large
English de_identify.rb deidentify_rb
English de_identify.rb_no_regex deidentify_rb_no_regex

Chunk resolvers

Language nlu.load() reference Spark NLP Model reference
English resolve_chunk.athena_conditions chunkresolve_athena_conditions_healthcare
English resolve_chunk.cpt_clinical chunkresolve_cpt_clinical
English resolve_chunk.icd10cm.clinical chunkresolve_icd10cm_clinical
English resolve_chunk.icd10cm.diseases_clinical chunkresolve_icd10cm_diseases_clinical
English resolve_chunk.icd10cm.hcc_clinical chunkresolve_icd10cm_hcc_clinical
English resolve_chunk.icd10cm.hcc_healthcare chunkresolve_icd10cm_hcc_healthcare
English resolve_chunk.icd10cm.injuries chunkresolve_icd10cm_injuries_clinical
English resolve_chunk.icd10cm.musculoskeletal chunkresolve_icd10cm_musculoskeletal_clinical
English resolve_chunk.icd10cm.neoplasms chunkresolve_icd10cm_neoplasms_clinical
English resolve_chunk.icd10cm.poison chunkresolve_icd10cm_poison_ext_clinical
English resolve_chunk.icd10cm.puerile chunkresolve_icd10cm_puerile_clinical
English resolve_chunk.icd10pcs.clinical chunkresolve_icd10pcs_clinical
English resolve_chunk.icdo.clinical chunkresolve_icdo_clinical
English resolve_chunk.loinc chunkresolve_loinc_clinical
English resolve_chunk.rxnorm.cd chunkresolve_rxnorm_cd_clinical
English resolve_chunk.rxnorm.in chunkresolve_rxnorm_in_clinical
English resolve_chunk.rxnorm.in_healthcare chunkresolve_rxnorm_in_healthcare
English resolve_chunk.rxnorm.sbd chunkresolve_rxnorm_sbd_clinical
English resolve_chunk.rxnorm.scd chunkresolve_rxnorm_scd_clinical
English resolve_chunk.rxnorm.scdc chunkresolve_rxnorm_scdc_clinical
English resolve_chunk.rxnorm.scdc_healthcare chunkresolve_rxnorm_scdc_healthcare
English resolve_chunk.rxnorm.xsmall.clinical chunkresolve_rxnorm_xsmall_clinical
English resolve_chunk.snomed.findings chunkresolve_snomed_findings_clinical

New Classifiers

Language nlu.load() reference Spark NLP Model reference
English classify.icd10.clinical classifier_icd10cm_hcc_clinical
English classify.icd10.healthcare classifier_icd10cm_hcc_healthcare
English classify.ade.biobert classifierdl_ade_biobert
English classify.ade.clinical classifierdl_ade_clinicalbert
English classify.ade.conversational classifierdl_ade_conversational_biobert
English classify.gender.biobert classifierdl_gender_biobert
English classify.gender.sbert classifierdl_gender_sbert
English classify.pico classifierdl_pico_biobert

German Medical models

nlu.load() reference Spark NLP Model reference
[embed] w2v_cc_300d
[embed.w2v] w2v_cc_300d
[resolve_chunk] chunkresolve_ICD10GM
[resolve_chunk.icd10gm] chunkresolve_ICD10GM
resolve_chunk.icd10gm.2021 chunkresolve_ICD10GM_2021
med_ner.legal ner_legal
med_ner ner_healthcare
med_ner.healthcare ner_healthcare
med_ner.healthcare_slim ner_healthcare_slim
med_ner.traffic ner_traffic

Spanish Medical models

nlu.load() reference Spark NLP Model reference
embed.scielo.150d embeddings_scielo_150d
embed.scielo.300d embeddings_scielo_300d
embed.scielo.50d embeddings_scielo_50d
embed.scielowiki.150d embeddings_scielowiki_150d
embed.scielowiki.300d embeddings_scielowiki_300d
embed.scielowiki.50d embeddings_scielowiki_50d
embed.sciwiki.150d embeddings_sciwiki_150d
embed.sciwiki.300d embeddings_sciwiki_300d
embed.sciwiki.50d embeddings_sciwiki_50d
med_ner ner_diag_proc
med_ner.neoplasm ner_neoplasms
med_ner.diag_proc ner_diag_proc

GPU Mode

You can now enable NLU GPU mode by setting gpu=true while loading a model. I.e. nlu.load('train.sentiment' gpu=True) . If must resart you kernel, if you already loaded a nlu pipeline withouth GPU mode.

Output Level Relation

This new output level is used for relation extractors and will give you 1 row per relation extracted.

Bug fixes

  • Fixed a bug that caused loading NLU models in offline mode not to work in some occasions

Install NLU in 1 line!

* Install NLU on Google Colab : !wget https://setup.johnsnowlabs.com/nlu/colab.sh  -O - | bash
* Install NLU via Pip         : ! pip install nlu pyspark==3.0.3

Additional NLU ressources

NLU Version 1.1.3

Intent and Action Classification, analyze Chinese News and the Crypto market, train a classifier that understands 100+ languages, translate between 200 + languages, answer questions, summarize text, and much more in NLU 1.1.3

We are very excited to announce that the latest NLU release comes with a new pretrained Intent Classifier and NER Action Extractor for text related to music, restaurants, and movies trained on the SNIPS dataset. Make sure to check out the models hub and the easy 1-liners for more info!

In addition to that, new NER and Embedding models for Bengali are now available

Finally, there is a new NLU Webinar with 9 accompanying tutorial notebooks which teach you a lot of things and is segmented into the following parts :

  • Part1: Easy 1 Liners
    • Spell checking/Sentiment/POS/NER/ BERTtology embeddings
  • Part2: Data analysis and NLP tasks on Crypto News Headline dataset
    • Preprocessing and extracting Emotions, Keywords, Named Entities and visualize them
  • Part3: NLU Multi-Lingual 1 Liners with Microsoft’s Marian Models
    • Translate between 200+ languages (and classify lang afterward)
  • Part 4: Data analysis and NLP tasks on Chinese News Article Dataset
    • Word Segmentation, Lemmatization, Extract Keywords, Named Entities and translate to english
  • Part 5: Train a sentiment Classifier that understands 100+ Languages
  • Part 6: Question answering, Summarization, Squad and more with Google’s T5

New Models

NLU 1.1.3 New Non-English Models

Language nlu.load() reference Spark NLP Model reference Type
Bengali bn.ner.cc_300d bengaliner_cc_300d NerDLModel
Bengali bn.embed bengali_cc_300d NerDLModel
Bengali bn.embed.cc_300d bengali_cc_300d Word Embeddings Model (Alias)
Bengali bn.embed.glove bengali_cc_300d Word Embeddings Model (Alias)

NLU 1.1.3 New English Models

Language nlu.load() reference Spark NLP Model reference Type
English en.classify.snips nerdl_snips_100d NerDLModel
English en.ner.snips classifierdl_use_snips ClassifierDLModel

New NLU Webinar

State-of-the-art Natural Language Processing for 200+ Languages with 1 Line of code

Talk Abstract

Learn to harness the power of 1,000+ production-grade & scalable NLP models for 200+ languages - all available with just 1 line of Python code by leveraging the open-source NLU library, which is powered by the widely popular Spark NLP.

John Snow Labs has delivered over 80 releases of Spark NLP to date, making it the most widely used NLP library in the enterprise and providing the AI community with state-of-the-art accuracy and scale for a variety of common NLP tasks. The most recent releases include pre-trained models for over 200 languages - including languages that do not use spaces for word segmentation algorithms like Chinese, Japanese, and Korean, and languages written from right to left like Arabic, Farsi, Urdu, and Hebrew. All software and models are free and open source under an Apache 2.0 license.

This webinar will show you how to leverage the multi-lingual capabilities of Spark NLP & NLU - including automated language detection for up to 375 languages, and the ability to perform translation, named entity recognition, stopword removal, lemmatization, and more in a variety of language families. We will create Python code in real-time and solve these problems in just 30 minutes. The notebooks will then be made freely available online.

You can watch the video here,

NLU 1.1.3 New Notebooks and tutorials

New Webinar Notebooks

  1. NLU basics, easy 1-liners (Spellchecking, sentiment, NER, POS, BERT
  2. Analyze Crypto News dataset with Keyword extraction, NER, Emotional distribution, and stemming
  3. Translate Crypto News dataset between 300 Languages with the Marian Model (German, French, Hebrew examples)
  4. Translate Crypto News dataset between 300 Languages with the Marian Model (Hindi, Russian, Chinese examples)
  5. Analyze Chinese News Headlines with Chinese Word Segmentation, Lemmatization, NER, and Keyword extraction
  6. Train a Sentiment Classifier that will understand 100+ languages on just a French Dataset with the powerful Language Agnostic Bert Embeddings
  7. Summarize text and Answer Questions with T5
  8. Solve any task in 1 line from SQUAD, GLUE and SUPER GLUE with T5
  9. Overview of models for various languages

New easy NLU 1-liners in NLU 1.1.3

nlu.load("en.classify.snips").predict("book a spot for nona gray  myrtle and alison at a top-rated brasserie that is distant from wilson av on nov  the 4th  2030 that serves ouzeri",output_level = "document")

outputs :

ner_confidence entities document Entities_Classes
[1.0, 1.0, 0.9997000098228455, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.9990000128746033, 1.0, 1.0, 1.0, 0.9965000152587891, 0.9998999834060669, 0.9567000269889832, 1.0, 1.0, 1.0, 0.9980000257492065, 0.9991999864578247, 0.9988999962806702, 1.0, 1.0, 0.9998999834060669] [‘nona gray myrtle and alison’, ‘top-rated’, ‘brasserie’, ‘distant’, ‘wilson av’, ‘nov the 4th 2030’, ‘ouzeri’] book a spot for nona gray myrtle and alison at a top-rated brasserie that is distant from wilson av on nov the 4th 2030 that serves ouzeri [‘party_size_description’, ‘sort’, ‘restaurant_type’, ‘spatial_relation’, ‘poi’, ‘timeRange’, ‘cuisine’]

Named Entity Recognition (NER) Model in Bengali (bengaliner_cc_300d)

# Bengali for: 'Iajuddin Ahmed passed Matriculation from Munshiganj High School in 1947 and Intermediate from Munshiganj Horganga College in 1950.'
nlu.load("bn.ner.cc_300d").predict("১৯৪৮ সালে ইয়াজউদ্দিন আহম্মেদ মুন্সিগঞ্জ উচ্চ বিদ্যালয় থেকে মেট্রিক পাশ করেন এবং ১৯৫০ সালে মুন্সিগঞ্জ হরগঙ্গা কলেজ থেকে ইন্টারমেডিয়েট পাশ করেন",output_level = "document")

outputs :

ner_confidence entities Entities_Classes document
[0.9987999796867371, 0.9854000210762024, 0.8604000210762024, 0.6686999797821045, 0.5289999842643738, 0.7009999752044678, 0.7684999704360962, 0.9979000091552734, 0.9976000189781189, 0.9930999875068665, 0.9994000196456909, 0.9879000186920166, 0.7407000064849854, 0.9215999841690063, 0.7657999992370605, 0.39419999718666077, 0.9124000072479248, 0.9932000041007996, 0.9919999837875366, 0.995199978351593, 0.9991999864578247] [‘সালে’, ‘ইয়াজউদ্দিন আহম্মেদ’, ‘মুন্সিগঞ্জ উচ্চ বিদ্যালয়’, ‘সালে’, ‘মুন্সিগঞ্জ হরগঙ্গা কলেজ’] [‘TIME’, ‘PER’, ‘ORG’, ‘TIME’, ‘ORG’] ১৯৪৮ সালে ইয়াজউদ্দিন আহম্মেদ মুন্সিগঞ্জ উচ্চ বিদ্যালয় থেকে মেট্রিক পাশ করেন এবং ১৯৫০ সালে মুন্সিগঞ্জ হরগঙ্গা কলেজ থেকে ইন্টারমেডিয়েট পাশ করেন

Identify intent in general text - SNIPS dataset

nlu.load("en.ner.snips").predict("I want to bring six of us to a bistro in town that serves hot chicken sandwich that is within the same area",output_level = "document")

outputs :

document snips snips_confidence
I want to bring six of us to a bistro in town that serves hot chicken sandwich that is within the same area BookRestaurant 1

Word Embeddings for Bengali (bengali_cc_300d)

# Bengali for : 'Iajuddin Ahmed passed Matriculation from Munshiganj High School in 1947 and Intermediate from Munshiganj Horganga College in 1950.'
nlu.load("bn.embed").predict("১৯৪৮ সালে ইয়াজউদ্দিন আহম্মেদ মুন্সিগঞ্জ উচ্চ বিদ্যালয় থেকে মেট্রিক পাশ করেন এবং ১৯৫০ সালে মুন্সিগঞ্জ হরগঙ্গা কলেজ থেকে ইন্টারমেডিয়েট পাশ করেন",output_level = "document")

outputs :

document bn_embed_embeddings
১৯৪৮ সালে ইয়াজউদ্দিন আহম্মেদ মুন্সিগঞ্জ উচ্চ বিদ্যালয় থেকে মেট্রিক পাশ করেন এবং ১৯৫০ সালে মুন্সিগঞ্জ হরগঙ্গা কলেজ থেকে ইন্টারমেডিয়েট পাশ করেন [-0.0828 0.0683 0.0215 … 0.0679 -0.0484…]

NLU 1.1.3 Enhancements

  • Added automatic conversion to Sentence Embeddings of Word Embeddings when there is no Sentence Embedding Avaiable and a model needs the converted version to run.

NLU 1.1.3 Bug Fixes

  • Fixed a bug that caused ur.sentiment NLU pipeline to build incorrectly
  • Fixed a bug that caused sentiment.imdb.glove NLU pipeline to build incorrectly
  • Fixed a bug that caused en.sentiment.glove.imdb NLU pipeline to build incorrectly
  • Fixed a bug that caused Spark 2.3.X environments to crash.

NLU Installation

# PyPi
!pip install nlu pyspark==2.4.7
#Conda
# Install NLU from Anaconda/Conda
conda install -os_components johnsnowlabs nlu

Additional NLU ressources

NLU Version 1.1.2

Hindi WordEmbeddings , Bengali Named Entity Recognition (NER), 30+ new models, analyze Crypto news with John Snow Labs NLU 1.1.2

We are very happy to announce NLU 1.1.2 has been released with the integration of 30+ models and pipelines Bengali Named Entity Recognition, Hindi Word Embeddings, and state-of-the-art transformer based OntoNotes models and pipelines from the incredible Spark NLP 2.7.3 Release in addition to a few bugfixes.
In addition to that, there is a new NLU Webinar video showcasing in detail how to use NLU to analyze a crypto news dataset to extract keywords unsupervised and predict sentimential/emotional distributions of the dataset and much more!

Python’s NLU library: 1,000+ models, 200+ Languages, State of the Art Accuracy, 1 Line of code - NLU NYC/DC NLP Meetup Webinar

Using just 1 line of Python code by leveraging the NLU library, which is powered by the award-winning Spark NLP.

This webinar covers, using live coding in real-time, how to deliver summarization, translation, unsupervised keyword extraction, emotion analysis, question answering, spell checking, named entity recognition, document classification, and other common NLP tasks. T his is all done with a single line of code, that works directly on Python strings or pandas data frames. Since NLU is based on Spark NLP, no code changes are required to scale processing to multi-core or cluster environment - integrating natively with Ray, Dask, or Spark data frames.

The recent releases for Spark NLP and NLU include pre-trained models for over 200 languages and language detection for 375 languages. This includes 20 languages families; non-Latin alphabets; languages that do not use spaces for word segmentation like Chinese, Japanese, and Korean; and languages written from right to left like Arabic, Farsi, Urdu, and Hebrew. We’ll also cover some of the algorithms and models that are included. The code notebooks will be freely available online.

NLU 1.1.2 New Non-English Models

Language nlu.load() reference Spark NLP Model reference Type
Bengali bn.ner ner_jifs_glove_840B_300d Word Embeddings Model (Alias)
Bengali bn.ner.glove ner_jifs_glove_840B_300d Word Embeddings Model (Alias)
Hindi hi.embed hindi_cc_300d NerDLModel
Bengali bn.lemma lemma Lemmatizer
Japanese ja.lemma lemma Lemmatizer
Bihari bh.lemma lemma Lemma
Amharic am.lemma lemma Lemma

NLU 1.1.2 New English Models and Pipelines

Language nlu.load() reference Spark NLP Model reference Type
English en.ner.onto.bert.small_l2_128 onto_small_bert_L2_128 NerDLModel
English en.ner.onto.bert.small_l4_256 onto_small_bert_L4_256 NerDLModel
English en.ner.onto.bert.small_l4_512 onto_small_bert_L4_512 NerDLModel
English en.ner.onto.bert.small_l8_512 onto_small_bert_L8_512 NerDLModel
English en.ner.onto.bert.cased_base onto_bert_base_cased NerDLModel
English en.ner.onto.bert.cased_large onto_bert_large_cased NerDLModel
English en.ner.onto.electra.uncased_small onto_electra_small_uncased NerDLModel
English en.ner.onto.electra.uncased_base onto_electra_base_uncased NerDLModel
English en.ner.onto.electra.uncased_large onto_electra_large_uncased NerDLModel
English en.ner.onto.bert.tiny onto_recognize_entities_bert_tiny Pipeline
English en.ner.onto.bert.mini onto_recognize_entities_bert_mini Pipeline
English en.ner.onto.bert.small onto_recognize_entities_bert_small Pipeline
English en.ner.onto.bert.medium onto_recognize_entities_bert_medium Pipeline
English en.ner.onto.bert.base onto_recognize_entities_bert_base Pipeline
English en.ner.onto.bert.large onto_recognize_entities_bert_large Pipeline
English en.ner.onto.electra.small onto_recognize_entities_electra_small Pipeline
English en.ner.onto.electra.base onto_recognize_entities_electra_base Pipeline
English en.ner.onto.large onto_recognize_entities_electra_large Pipeline

New Tutorials and Notebooks

NLU 1.1.2 Bug Fixes

  • Fixed a bug that caused NER confidences not beeing extracted
  • Fixed a bug that caused nlu.load(‘spell’) to crash
  • Fixed a bug that caused Uralic/Estonian/ET language models not to be loaded properly

New Easy NLU 1-liners in 1.1.2

Named Entity Recognition for Bengali (GloVe 840B 300d)

#Bengali for :  It began to be widely used in the United States in the early '90s.
nlu.load("bn.ner").predict("৯০ এর দশকের শুরুর দিকে বৃহৎ আকারে মার্কিন যুক্তরাষ্ট্রে এর প্রয়োগের প্রক্রিয়া শুরু হয়'")

output :

entities token Entities_classes ner_confidence
[‘মার্কিন যুক্তরাষ্ট্রে’] ৯০ [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] এর [‘LOC’] 0.9999
[‘মার্কিন যুক্তরাষ্ট্রে’] দশকের [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] শুরুর [‘LOC’] 0.9969
[‘মার্কিন যুক্তরাষ্ট্রে’] দিকে [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] বৃহৎ [‘LOC’] 0.9994
[‘মার্কিন যুক্তরাষ্ট্রে’] আকারে [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] মার্কিন [‘LOC’] 0.9602
[‘মার্কিন যুক্তরাষ্ট্রে’] যুক্তরাষ্ট্রে [‘LOC’] 0.4134
[‘মার্কিন যুক্তরাষ্ট্রে’] এর [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] প্রয়োগের [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] প্রক্রিয়া [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] শুরু [‘LOC’] 0.9999
[‘মার্কিন যুক্তরাষ্ট্রে’] হয় [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] [‘LOC’] 1

Bengali Lemmatizer

#Bengali for :  One morning in the marble-decorated building of Vaidyanatha, an obese monk was engaged in the enchantment of Duis and the milk service of one and a half Vaidyanatha. Give me two to eat
nlu.load("bn.lemma").predict("একদিন প্রাতে বৈদ্যনাথের মার্বলমণ্ডিত দালানে একটি স্থূলোদর সন্ন্যাসী দুইসের মোহনভোগ এবং দেড়সের দুগ্ধ সেবায় নিযুক্ত আছে বৈদ্যনাথ গায়ে একখানি চাদর দিয়া জোড়করে একান্ত বিনীতভাবে ভূতলে বসিয়া ভক্তিভরে পবিত্র ভোজনব্যাপার নিরীক্ষণ করিতেছিলেন এমন সময় কোনোমতে দ্বারীদের দৃষ্টি এড়াইয়া জীর্ণদেহ বালক সহিত একটি অতি শীর্ণকায়া রমণী গৃহে প্রবেশ করিয়া ক্ষীণস্বরে কহিল বাবু দুটি খেতে দাও")

output :

lemma document
[‘একদিন’, ‘প্রাতঃ’, ‘বৈদ্যনাথ’, ‘মার্বলমণ্ডিত’, ‘দালান’, ‘এক’, ‘স্থূলউদর’, ‘সন্ন্যাসী’, ‘দুইসের’, ‘মোহনভোগ’, ‘এবং’, ‘দেড়সের’, ‘দুগ্ধ’, ‘সেবা’, ‘নিযুক্ত’, ‘আছে’, ‘বৈদ্যনাথ’, ‘গা’, ‘একখান’, ‘চাদর’, ‘দেওয়া’, ‘জোড়কর’, ‘একান্ত’, ‘বিনীতভাব’, ‘ভূতল’, ‘বসা’, ‘ভক্তিভরা’, ‘পবিত্র’, ‘ভোজনব্যাপার’, ‘নিরীক্ষণ’, ‘করা’, ‘এমন’, ‘সময়’, ‘কোনোমত’, ‘দ্বারী’, ‘দৃষ্টি’, ‘এড়ানো’, ‘জীর্ণদেহ’, ‘বালক’, ‘সহিত’, ‘এক’, ‘অতি’, ‘শীর্ণকায়া’, ‘রমণী’, ‘গৃহ’, ‘প্রবেশ’, ‘বিশ্বাস’, ‘ক্ষীণস্বর’, ‘কহা’, ‘বাবু’, ‘দুই’, ‘খাওয়া’, ‘দাওয়া’] একদিন প্রাতে বৈদ্যনাথের মার্বলমণ্ডিত দালানে একটি স্থূলোদর সন্ন্যাসী দুইসের মোহনভোগ এবং দেড়সের দুগ্ধ সেবায় নিযুক্ত আছে বৈদ্যনাথ গায়ে একখানি চাদর দিয়া জোড়করে একান্ত বিনীতভাবে ভূতলে বসিয়া ভক্তিভরে পবিত্র ভোজনব্যাপার নিরীক্ষণ করিতেছিলেন এমন সময় কোনোমতে দ্বারীদের দৃষ্টি এড়াইয়া জীর্ণদেহ বালক সহিত একটি অতি শীর্ণকায়া রমণী গৃহে প্রবেশ করিয়া ক্ষীণস্বরে কহিল বাবু দুটি খেতে দাও

Japanese Lemmatizer

#Japanese for :  Some residents were uncomfortable with this, but it seems that no one is now openly protesting or protesting.
nlu.load("ja.lemma").predict("これに不快感を示す住民はいましたが,現在,表立って反対や抗議の声を挙げている住民はいないようです。")

output :

lemma document
[‘これ’, ‘にる’, ‘不快’, ‘感’, ‘を’, ‘示す’, ‘住民’, ‘はる’, ‘いる’, ‘まする’, ‘たる’, ‘がる’, ‘,’, ‘現在’, ‘,’, ‘表立つ’, ‘てる’, ‘反対’, ‘やる’, ‘抗議’, ‘のる’, ‘声’, ‘を’, ‘挙げる’, ‘てる’, ‘いる’, ‘住民’, ‘はる’, ‘いる’, ‘なぐ’, ‘よう’, ‘です’, ‘。’] これに不快感を示す住民はいましたが,現在,表立って反対や抗議の声を挙げている住民はいないようです。

Amharic Lemmatizer

#Aharic for :  Bookmark the permalink.
nlu.load("am.lemma").predict("መጽሐፉን መጽሐፍ ኡ ን አስያዛት አስያዝ ኧ ኣት ።")

output :

lemma document
[‘’, ‘መጽሐፍ’, ‘ኡ’, ‘ን’, ‘’, ‘አስያዝ’, ‘ኧ’, ‘ኣት’, ‘።’] መጽሐፉን መጽሐፍ ኡ ን አስያዛት አስያዝ ኧ ኣት ።

Bhojpuri Lemmatizer

#Bhojpuri for : In this event, participation of World Bhojpuri Conference, Purvanchal Ekta Manch, Veer Kunwar Singh Foundation, Purvanchal Bhojpuri Mahasabha, and Herf - Media.
nlu.load("bh.lemma").predict("एह आयोजन में विश्व भोजपुरी सम्मेलन , पूर्वांचल एकता मंच , वीर कुँवर सिंह फाउन्डेशन , पूर्वांचल भोजपुरी महासभा , अउर हर्फ - मीडिया के सहभागिता बा ।")

output :

lemma document
[‘एह’, ‘आयोजन’, ‘में’, ‘विश्व’, ‘भोजपुरी’, ‘सम्मेलन’, ‘COMMA’, ‘पूर्वांचल’, ‘एकता’, ‘मंच’, ‘COMMA’, ‘वीर’, ‘कुँवर’, ‘सिंह’, ‘फाउन्डेशन’, ‘COMMA’, ‘पूर्वांचल’, ‘भोजपुरी’, ‘महासभा’, ‘COMMA’, ‘अउर’, ‘हर्फ’, ‘-‘, ‘मीडिया’, ‘को’, ‘सहभागिता’, ‘बा’, ‘।’] एह आयोजन में विश्व भोजपुरी सम्मेलन , पूर्वांचल एकता मंच , वीर कुँवर सिंह फाउन्डेशन , पूर्वांचल भोजपुरी महासभा , अउर हर्फ - मीडिया के सहभागिता बा ।

Named Entity Recognition - BERT Tiny (OntoNotes)

nlu.load("en.ner.onto.bert.small_l2_128").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.8536999821662903, 0.7195000052452087, 0.746…] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘1970s’, ‘1980s’, ‘Seattle’, ‘Washington’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’]

Named Entity Recognition - BERT Mini (OntoNotes)

nlu.load("en.ner.onto.bert.small_l4_256").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.835099995136261, 0.40450000762939453, 0.331…] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘1970s and 1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘ORG’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘GPE’, ‘GPE’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’]

Named Entity Recognition - BERT Small (OntoNotes)

nlu.load("en.ner.onto.bert.small_l4_512").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.964900016784668, 0.8299000263214111, 0.9607…] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘the 1970s and 1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’]

Named Entity Recognition - BERT Medium (OntoNotes)

nlu.load("en.ner.onto.bert.small_l8_512").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.916700005531311, 0.5873000025749207, 0.8816…] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘the 1970s and 1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’]

Named Entity Recognition - BERT Base (OntoNotes)

nlu.load("en.ner.onto.bert.cased_base").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.504800021648407, 0.47290000319480896, 0.462…] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘the 1970s and 1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’]

Named Entity Recognition - BERT Large (OntoNotes)

nlu.load("en.ner.onto.electra.uncased_small").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.7213000059127808, 0.6384000182151794, 0.731…] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘1970s’, ‘1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’]

Named Entity Recognition - ELECTRA Small (OntoNotes)

nlu.load("en.ner.onto.electra.uncased_small").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence Entities_classes entities
[0.8496000170707703, 0.4465999901294708, 0.568…] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘1970s’, ‘1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’]

Named Entity Recognition - ELECTRA Base (OntoNotes)

nlu.load("en.ner.onto.electra.uncased_base").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadellabase.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.5134000182151794, 0.9419000148773193, 0.802…] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘the 1970s’, ‘1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’]

Named Entity Recognition - ELECTRA Large (OntoNotes)


nlu.load("en.ner.onto.electra.uncased_large").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadellabase.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.8442000150680542, 0.26840001344680786, 0.57…] [‘William Henry Gates’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘1970s’, ‘1980s’, ‘Seattle’, ‘Washington’, ‘Gates co-founded’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’, ‘largest’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’, ‘GPE’]

Recognize Entities OntoNotes - BERT Tiny


nlu.load("en.ner.onto.bert.tiny").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.994700014591217, 0.9412999749183655, 0.9685…] [‘Johnson’, ‘first’, ‘2001’, ‘Parliament’, ‘eight years’, ‘London’, ‘2008 to 2016’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘ORG’, ‘DATE’, ‘GPE’, ‘DATE’]

Recognize Entities OntoNotes - BERT Mini

nlu.load("en.ner.onto.bert.mini").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.996399998664856, 0.9733999967575073, 0.8766…] [‘Johnson’, ‘first’, ‘2001’, ‘eight years’, ‘London’, ‘2008 to 2016’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘DATE’]

Recognize Entities OntoNotes - BERT Small

nlu.load("en.ner.onto.bert.small").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.9987999796867371, 0.9610000252723694, 0.998…] [‘Johnson’, ‘first’, ‘2001’, ‘eight years’, ‘London’, ‘2008 to 2016’, ‘Parliament’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘DATE’, ‘ORG’]

Recognize Entities OntoNotes - BERT Medium


nlu.load("en.ner.onto.bert.medium").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.9969000220298767, 0.8575999736785889, 0.995…] [‘Johnson’, ‘first’, ‘2001’, ‘eight years’, ‘London’, ‘2008 to 2016’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘DATE’]

Recognize Entities OntoNotes - BERT Base

nlu.load("en.ner.onto.bert.base").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.996999979019165, 0.933899998664856, 0.99930…] [‘Johnson’, ‘first’, ‘2001’, ‘Parliament’, ‘eight years’, ‘London’, ‘2008 to 2016’, ‘Parliament’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘ORG’, ‘DATE’, ‘GPE’, ‘DATE’, ‘ORG’]

Recognize Entities OntoNotes - BERT Large

nlu.load("en.ner.onto.bert.large").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.9786999821662903, 0.9549000263214111, 0.998…] [‘Johnson’, ‘first’, ‘2001’, ‘Parliament’, ‘eight years’, ‘London’, ‘2008 to 2016’, ‘Parliament’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘ORG’, ‘DATE’, ‘GPE’, ‘DATE’, ‘ORG’]

Recognize Entities OntoNotes - ELECTRA Small

nlu.load("en.ner.onto.electra.small").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.9952999949455261, 0.8589000105857849, 0.996…] [‘Johnson’, ‘first’, ‘2001’, ‘eight years’, ‘London’, ‘2008 to 2016’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘DATE’]

Recognize Entities OntoNotes - ELECTRA Base

nlu.load("en.ner.onto.electra.base").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.9987999796867371, 0.9474999904632568, 0.999…] [‘Johnson’, ‘first’, ‘2001’, ‘Parliament’, ‘eight years’, ‘London’, ‘2008’, ‘2016’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘ORG’, ‘DATE’, ‘GPE’, ‘DATE’, ‘DATE’]

Recognize Entities OntoNotes - ELECTRA Large

nlu.load("en.ner.onto.large").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.9998000264167786, 0.9613999724388123, 0.998…] [‘Johnson’, ‘first’, ‘2001’, ‘eight years’, ‘London’, ‘2008 to 2016’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘DATE’]

NLU Installation

# PyPi
!pip install nlu pyspark==2.4.7
#Conda
# Install NLU from Anaconda/Conda
conda install -os_components johnsnowlabs nlu

Additional NLU ressources

NLU Version 1.1.1

We are very excited to release NLU 1.1.1! This release features 3 new tutorial notebooks for Open/Closed book question answering with Google’s T5, Intent classification and Aspect Based NER. In Addition NLU 1.1.0 comes with 25+ pretrained models and pipelines in Amharic, Bengali, Bhojpuri, Japanese, and Korean languages from the amazing Spark2.7.2 release Finally NLU now supports running on Spark 2.3 clusters.

NLU 1.1.1 New Non-English Models

Language nlu.load() reference Spark NLP Model reference Type
Arabic ar.ner arabic_w2v_cc_300d Named Entity Recognizer
Arabic ar.embed.aner aner_cc_300d Word Embedding
Arabic ar.embed.aner.300d aner_cc_300d Word Embedding (Alias)
Bengali bn.stopwords stopwords_bn Stopwords Cleaner
Bengali bn.pos pos_msri Part of Speech
Thai th.segment_words wordseg_best Word Segmenter
Thai th.pos pos_lst20 Part of Speech
Thai th.sentiment sentiment_jager_use Sentiment Classifier
Thai th.classify.sentiment sentiment_jager_use Sentiment Classifier (Alias)
Chinese zh.pos.ud_gsd_trad pos_ud_gsd_trad Part of Speech
Chinese zh.segment_words.gsd wordseg_gsd_ud_trad Word Segmenter
Bihari bh.pos pos_ud_bhtb Part of Speech
Amharic am.pos pos_ud_att Part of Speech

NLU 1.1.1 New English Models and Pipelines

Language nlu.load() reference Spark NLP Model reference Type
English en.sentiment.glove analyze_sentimentdl_glove_imdb Sentiment Classifier
English en.sentiment.glove.imdb analyze_sentimentdl_glove_imdb Sentiment Classifier (Alias)
English en.classify.sentiment.glove.imdb analyze_sentimentdl_glove_imdb Sentiment Classifier (Alias)
English en.classify.sentiment.glove analyze_sentimentdl_glove_imdb Sentiment Classifier (Alias)
English en.classify.trec50.pipe classifierdl_use_trec50_pipeline Language Classifier
English en.ner.onto.large onto_recognize_entities_electra_large Named Entity Recognizer
English en.classify.questions.atis classifierdl_use_atis Intent Classifier
English en.classify.questions.airline classifierdl_use_atis Intent Classifier (Alias)
English en.classify.intent.atis classifierdl_use_atis Intent Classifier (Alias)
English en.classify.intent.airline classifierdl_use_atis Intent Classifier (Alias)
English en.ner.atis nerdl_atis_840b_300d Aspect based NER
English en.ner.airline nerdl_atis_840b_300d Aspect based NER (Alias)
English en.ner.aspect.airline nerdl_atis_840b_300d Aspect based NER (Alias)
English en.ner.aspect.atis nerdl_atis_840b_300d Aspect based NER (Alias)

New Easy NLU 1-liner Examples :

Extract aspects and entities from airline questions (ATIS dataset)

	
nlu.load("en.ner.atis").predict("i want to fly from baltimore to dallas round trip")
output:  ["baltimore"," dallas", "round trip"]

Intent Classification for Airline Traffic Information System queries (ATIS dataset)


nlu.load("en.classify.questions.atis").predict("what is the price of flight from newyork to washington")
output:  "atis_airfare"	

Recognize Entities OntoNotes - ELECTRA Large


nlu.load("en.ner.onto.large").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London.")	
output:  ["Johnson", "first", "2001", "eight years", "London"]	

Question classification of open-domain and fact-based questions Pipeline - TREC50

nlu.load("en.classify.trec50.component_list").predict("When did the construction of stone circles begin in the UK? ")
output:  LOC_other

Traditional Chinese Word Segmentation

# 'However, this treatment also creates some problems' in Chinese
nlu.load("zh.segment_words.gsd").predict("然而,這樣的處理也衍生了一些問題。")
output:  ["然而",",","這樣","的","處理","也","衍生","了","一些","問題","。"]

Part of Speech for Traditional Chinese

# 'However, this treatment also creates some problems' in Chinese
nlu.load("zh.pos.ud_gsd_trad").predict("然而,這樣的處理也衍生了一些問題。")

Output:

Token POS
然而 ADV
PUNCT
這樣 PRON
PART
處理 NOUN
ADV
衍生 VERB
PART
一些 ADJ
問題 NOUN
PUNCT

Thai Word Segment Recognition

# 'Mona Lisa is a 16th-century oil painting created by Leonardo held at the Louvre in Paris' in Thai
nlu.loadnlu.load("th.segment_words").predict("Mona Lisa เป็นภาพวาดสีน้ำมันในศตวรรษที่ 16 ที่สร้างโดย Leonardo จัดขึ้นที่พิพิธภัณฑ์ลูฟร์ในปารีส")

Output:

token
M
o
n
a
Lisa
เป็น
ภาพ
สีน้ำ
มัน
ใน
ศตวรรษ
ที่
16
ที่
สร้าง
L
e
o
n
a
r
d
o
จัด
ขึ้น
ที่
พิพิธภัณฑ์
ลูฟร์
ใน
ปารีส

Part of Speech for Bengali (POS)

# 'The village is also called 'Mod' in Tora language' in Behgali 
nlu.load("bn.pos").predict("বাসস্থান-ঘরগৃহস্থালি তোড়া ভাষায় গ্রামকেও বলে ` মোদ ' ৷")

Output:

token pos
বাসস্থান-ঘরগৃহস্থালি NN
তোড়া NNP
ভাষায় NN
গ্রামকেও NN
বলে VM
` SYM
মোদ NN
SYM
SYM

Stop Words Cleaner for Bengali

# 'This language is not enough' in Bengali 
df = nlu.load("bn.stopwords").predict("এই ভাষা যথেষ্ট নয়")

Output:

cleanTokens token
ভাষা এই
যথেষ্ট ভাষা
নয় যথেষ্ট
None নয়

Part of Speech for Bengali


# 'The people of Ohu know that the foundation of Bhojpuri was shaken' in Bengali
nlu.load('bh.pos').predict("ओहु लोग के मालूम बा कि श्लील होखते भोजपुरी के नींव हिल जाई")

Output:

pos token
DET ओहु
NOUN लोग
ADP के
NOUN मालूम
VERB बा
SCONJ कि
ADJ श्लील
VERB होखते
PROPN भोजपुरी
ADP के
NOUN नींव
VERB हिल
AUX जाई

Amharic Part of Speech (POS)

# ' "Son, finish the job," he said.' in Amharic
nlu.load('am.pos').predict('ልጅ ኡ ን ሥራ ው ን አስጨርስ ኧው ኣል ኧሁ ።"')

Output:

pos token
NOUN ልጅ
DET
PART
NOUN ሥራ
DET
PART
VERB አስጨርስ
PRON ኧው
AUX ኣል
PRON ኧሁ
PUNCT
NOUN

Thai Sentiment Classification

#  'I love peanut butter and jelly!' in thai
nlu.load('th.classify.sentiment').predict('ฉันชอบเนยถั่วและเยลลี่!')[['sentiment','sentiment_confidence']]

Output:

sentiment sentiment_confidence
positive 0.999998

Arabic Named Entity Recognition (NER)

# 'In 1918, the forces of the Arab Revolt liberated Damascus with the help of the British' in Arabic
nlu.load('ar.ner').predict('في عام 1918 حررت قوات الثورة العربية دمشق بمساعدة من الإنكليز',output_level='chunk')[['entities_confidence','ner_confidence','entities']]

Output:

entity_class ner_confidence entities
ORG [1.0, 1.0, 1.0, 0.9997000098228455, 0.9840999841690063, 0.9987999796867371, 0.9990000128746033, 0.9998999834060669, 0.9998999834060669, 0.9993000030517578, 0.9998999834060669] قوات الثورة العربية
LOC [1.0, 1.0, 1.0, 0.9997000098228455, 0.9840999841690063, 0.9987999796867371, 0.9990000128746033, 0.9998999834060669, 0.9998999834060669, 0.9993000030517578, 0.9998999834060669] دمشق
PER [1.0, 1.0, 1.0, 0.9997000098228455, 0.9840999841690063, 0.9987999796867371, 0.9990000128746033, 0.9998999834060669, 0.9998999834060669, 0.9993000030517578, 0.9998999834060669] الإنكليز

NLU 1.1.1 Enhancements :

  • Spark 2.3 compatibility

New NLU Notebooks and Tutorials

Installation

# PyPi
!pip install nlu pyspark==2.4.7
#Conda
# Install NLU from Anaconda/Conda
conda install -os_components johnsnowlabs nlu

Additional NLU ressources

NLU Version 1.1.0

We are incredibly excited to release NLU 1.1.0! This release integrates the 720+ new models from the latest Spark-NLP 2.7.0 + releases. You can now achieve state-of-the-art results with Sequence2Sequence transformers for problems like text summarization, question answering, translation between 192+ languages and extract Named Entity in various Right to Left written languages like Korean, Japanese, Chinese and many more in 1 line of code!
These new features are possible because of the integration of the Google’s T5 models and Microsoft’s Marian models transformers

NLU 1.1.0 has over 720+ new pretrained models and pipelines while extending the support of multi-lingual models to 192+ languages such as Chinese, Japanese, Korean, Arabic, Persian, Urdu, and Hebrew.

NLU 1.1.0 New Features

  • 720+ new models you can find an overview of all NLU models here and further documentation in the models hub
  • NEW: Introducing MarianTransformer annotator for machine translation based on MarianNMT models. Marian is an efficient, free Neural Machine Translation framework mainly being developed by the Microsoft Translator team (646+ pretrained models & pipelines in 192+ languages)
  • NEW: Introducing T5Transformer annotator for Text-To-Text Transfer Transformer (Google T5) models to achieve state-of-the-art results on multiple NLP tasks such as Translation, Summarization, Question Answering, Sentence Similarity, and so on
  • NEW: Introducing brand new and refactored language detection and identification models. The new LanguageDetectorDL is faster, more accurate, and supports up to 375 languages
  • NEW: Introducing WordSegmenter model for word segmentation of languages without any rule-based tokenization such as Chinese, Japanese, or Korean
  • NEW: Introducing DocumentNormalizer component for cleaning content from HTML or XML documents, applying either data cleansing using an arbitrary number of custom regular expressions either data extraction following the different parameters

NLU 1.1.0 New Notebooks, Tutorials and Articles

NLU 1.1.0 New Training Tutorials

Binary Classifier training Jupyter tutorials

Multi Class text Classifier training Jupyter tutorials

NLU 1.1.0 New Medium Tutorials

Translation

Translation example
You can translate between more than 192 Languages pairs with the Marian Models You need to specify the language your data is in as start_language and the language you want to translate to as target_language.
The language references must be ISO language codes

nlu.load('<start_language>.translate.<target_language>')

Translate Turkish to English:
nlu.load('tr.translate_to.fr')

Translate English to French:
nlu.load('en.translate_to.fr')

Translate French to Hebrew:
nlu.load('en.translate_to.fr')

translate_pipe = nlu.load('en.translate_to.fr')
df = translate_pipe.predict('Billy likes to go to the mall every sunday')
df
sentence translation
Billy likes to go to the mall every sunday Billy geht gerne jeden Sonntag ins Einkaufszentrum

Overview of every task available with T5

The T5 model is trained on various datasets for 17 different tasks which fall into 8 categories.

  1. Text summarization
  2. Question answering
  3. Translation
  4. Sentiment analysis
  5. Natural Language inference
  6. Coreference resolution
  7. Sentence Completion
  8. Word sense disambiguation

Every T5 Task with explanation:

Task Name Explanation
1.CoLA Classify if a sentence is gramaticaly correct
2.RTE Classify whether if a statement can be deducted from a sentence
3.MNLI Classify for a hypothesis and premise whether they contradict or contradict each other or neither of both (3 class).
4.MRPC Classify whether a pair of sentences is a re-phrasing of each other (semantically equivalent)
5.QNLI Classify whether the answer to a question can be deducted from an answer candidate.
6.QQP Classify whether a pair of questions is a re-phrasing of each other (semantically equivalent)
7.SST2 Classify the sentiment of a sentence as positive or negative
8.STSB Classify the sentiment of a sentence on a scale from 1 to 5 (21 Sentiment classes)
9.CB Classify for a premise and a hypothesis whether they contradict each other or not (binary).
10.COPA Classify for a question, premise, and 2 choices which choice the correct choice is (binary).
11.MultiRc Classify for a question, a paragraph of text, and an answer candidate, if the answer is correct (binary),
12.WiC Classify for a pair of sentences and a disambigous word if the word has the same meaning in both sentences.
13.WSC/DPR Predict for an ambiguous pronoun in a sentence what it is referring to.
14.Summarization Summarize text into a shorter representation.
15.SQuAD Answer a question for a given context.
16.WMT1. Translate English to German
17.WMT2. Translate English to French
18.WMT3. Translate English to Romanian

refer to this notebook to see how to use every T5 Task.

Question Answering

Question answering example)

Predict an answer to a question based on input context.
This is based on SQuAD - Context based question answering

Predicted Answer Question Context
carbon monoxide What does increased oxygen concentrations in the patient’s lungs displace? Hyperbaric (high-pressure) medicine uses special oxygen chambers to increase the partial pressure of O 2 around the patient and, when needed, the medical staff. Carbon monoxide poisoning, gas gangrene, and decompression sickness (the ’bends’) are sometimes treated using these devices. Increased O 2 concentration in the lungs helps to displace carbon monoxide from the heme group of hemoglobin. Oxygen gas is poisonous to the anaerobic bacteria that cause gas gangrene, so increasing its partial pressure helps kill them. Decompression sickness occurs in divers who decompress too quickly after a dive, resulting in bubbles of inert gas, mostly nitrogen and helium, forming in their blood. Increasing the pressure of O 2 as soon as possible is part of the treatment.
pie What did Joey eat for breakfast? Once upon a time, there was a squirrel named Joey. Joey loved to go outside and play with his cousin Jimmy. Joey and Jimmy played silly games together, and were always laughing. One day, Joey and Jimmy went swimming together 50 at their Aunt Julie’s pond. Joey woke up early in the morning to eat some food before they left. Usually, Joey would eat cereal, fruit (a pear), or oatmeal for breakfast. After he ate, he and Jimmy went to the pond. On their way there they saw their friend Jack Rabbit. They dove into the water and swam for several hours. The sun was out, but the breeze was cold. Joey and Jimmy got out of the water and started walking home. Their fur was wet, and the breeze chilled them. When they got home, they dried off, and Jimmy put on his favorite purple shirt. Joey put on a blue shirt with red and green dots. The two squirrels ate some food that Joey’s mom, Jasmine, made and went off to bed,’
# Set the task on T5
t5['t5'].setTask('question ') 


# define Data, add additional tags between sentences
data = ['''
What does increased oxygen concentrations in the patient’s lungs displace? 
context: Hyperbaric (high-pressure) medicine uses special oxygen chambers to increase the partial pressure of O 2 around the patient and, when needed, the medical staff. Carbon monoxide poisoning, gas gangrene, and decompression sickness (the ’bends’) are sometimes treated using these devices. Increased O 2 concentration in the lungs helps to displace carbon monoxide from the heme group of hemoglobin. Oxygen gas is poisonous to the anaerobic bacteria that cause gas gangrene, so increasing its partial pressure helps kill them. Decompression sickness occurs in divers who decompress too quickly after a dive, resulting in bubbles of inert gas, mostly nitrogen and helium, forming in their blood. Increasing the pressure of O 2 as soon as possible is part of the treatment.
''']


#Predict on text data with T5
t5.predict(data)

How to configure T5 task parameter for Squad Context based question answering and pre-process data

.setTask('question:) and prefix the context which can be made up of multiple sentences with context:

Example pre-processed input for T5 Squad Context based question answering:

question: What does increased oxygen concentrations in the patient’s lungs displace? 
context: Hyperbaric (high-pressure) medicine uses special oxygen chambers to increase the partial pressure of O 2 around the patient and, when needed, the medical staff. Carbon monoxide poisoning, gas gangrene, and decompression sickness (the ’bends’) are sometimes treated using these devices. Increased O 2 concentration in the lungs helps to displace carbon monoxide from the heme group of hemoglobin. Oxygen gas is poisonous to the anaerobic bacteria that cause gas gangrene, so increasing its partial pressure helps kill them. Decompression sickness occurs in divers who decompress too quickly after a dive, resulting in bubbles of inert gas, mostly nitrogen and helium, forming in their blood. Increasing the pressure of O 2 as soon as possible is part of the treatment.

Text Summarization

Summarization example

Summarizes a paragraph into a shorter version with the same semantic meaning, based on Text summarization

# Set the task on T5
pipe = nlu.load('summarize')

# define Data, add additional tags between sentences
data = [
'''
The belgian duo took to the dance floor on monday night with some friends . manchester united face newcastle in the premier league on wednesday . red devils will be looking for just their second league away win in seven . louis van gaal’s side currently sit two points clear of liverpool in fourth .
''',
'''  Calculus, originally called infinitesimal calculus or "the calculus of infinitesimals", is the mathematical study of continuous change, in the same way that geometry is the study of shape and algebra is the study of generalizations of arithmetic operations. It has two major branches, differential calculus and integral calculus; the former concerns instantaneous rates of change, and the slopes of curves, while integral calculus concerns accumulation of quantities, and areas under or between curves. These two branches are related to each other by the fundamental theorem of calculus, and they make use of the fundamental notions of convergence of infinite sequences and infinite series to a well-defined limit.[1] Infinitesimal calculus was developed independently in the late 17th century by Isaac Newton and Gottfried Wilhelm Leibniz.[2][3] Today, calculus has widespread uses in science, engineering, and economics.[4] In mathematics education, calculus denotes courses of elementary mathematical analysis, which are mainly devoted to the study of functions and limits. The word calculus (plural calculi) is a Latin word, meaning originally "small pebble" (this meaning is kept in medicine – see Calculus (medicine)). Because such pebbles were used for calculation, the meaning of the word has evolved and today usually means a method of computation. It is therefore used for naming specific methods of calculation and related theories, such as propositional calculus, Ricci calculus, calculus of variations, lambda calculus, and process calculus.'''
]


#Predict on text data with T5
pipe.predict(data)
Predicted summary Text
manchester united face newcastle in the premier league on wednesday . louis van gaal’s side currently sit two points clear of liverpool in fourth . the belgian duo took to the dance floor on monday night with some friends . the belgian duo took to the dance floor on monday night with some friends . manchester united face newcastle in the premier league on wednesday . red devils will be looking for just their second league away win in seven . louis van gaal’s side currently sit two points clear of liverpool in fourth .

Binary Sentence similarity/ Paraphrasing

Binary sentence similarity example Classify whether one sentence is a re-phrasing or similar to another sentence
This is a sub-task of GLUE and based on MRPC - Binary Paraphrasing/ sentence similarity classification

t5 = nlu.load('en.t5.base')
# Set the task on T5
t5['t5'].setTask('mrpc ')

# define Data, add additional tags between sentences
data = [
''' sentence1: We acted because we saw the existing evidence in a new light , through the prism of our experience on 11 September , " Rumsfeld said .
sentence2: Rather , the US acted because the administration saw " existing evidence in a new light , through the prism of our experience on September 11 "
'''
,
'''  
sentence1: I like to eat peanutbutter for breakfast
sentence2: 	I like to play football.
'''
]

#Predict on text data with T5
t5.predict(data)

| Sentence1 | Sentence2 | prediction| |————|————|———-| |We acted because we saw the existing evidence in a new light , through the prism of our experience on 11 September , “ Rumsfeld said .| Rather , the US acted because the administration saw “ existing evidence in a new light , through the prism of our experience on September 11 “ . | equivalent | | I like to eat peanutbutter for breakfast| I like to play football | not_equivalent |

How to configure T5 task for MRPC and pre-process text

.setTask('mrpc sentence1:) and prefix second sentence with sentence2:

Example pre-processed input for T5 MRPC - Binary Paraphrasing/ sentence similarity

mrpc 
sentence1: We acted because we saw the existing evidence in a new light , through the prism of our experience on 11 September , " Rumsfeld said . 
sentence2: Rather , the US acted because the administration saw " existing evidence in a new light , through the prism of our experience on September 11",

Regressive Sentence similarity/ Paraphrasing

Measures how similar two sentences are on a scale from 0 to 5 with 21 classes representing a regressive label.
This is a sub-task of GLUE and based onSTSB - Regressive semantic sentence similarity .

t5 = nlu.load('en.t5.base')
# Set the task on T5
t5['t5'].setTask('stsb ') 

# define Data, add additional tags between sentences
data = [
             
              ''' sentence1:  What attributes would have made you highly desirable in ancient Rome?  
                  sentence2:  How I GET OPPERTINUTY TO JOIN IT COMPANY AS A FRESHER?'
              '''
             ,
             '''  
              sentence1: What was it like in Ancient rome?
              sentence2: 	What was Ancient rome like?
              ''',
              '''  
              sentence1: What was live like as a King in Ancient Rome??
              sentence2: 	What was Ancient rome like?
              '''

             ]



#Predict on text data with T5
t5.predict(data)

Question1 Question2 prediction
What attributes would have made you highly desirable in ancient Rome? How I GET OPPERTINUTY TO JOIN IT COMPANY AS A FRESHER? 0
What was it like in Ancient rome? What was Ancient rome like? 5.0
What was live like as a King in Ancient Rome?? What is it like to live in Rome? 3.2

How to configure T5 task for stsb and pre-process text

.setTask('stsb sentence1:) and prefix second sentence with sentence2:

Example pre-processed input for T5 STSB - Regressive semantic sentence similarity

stsb
sentence1: What attributes would have made you highly desirable in ancient Rome?        
sentence2: How I GET OPPERTINUTY TO JOIN IT COMPANY AS A FRESHER?',

Grammar Checking

Grammar checking with T5 example) Judges if a sentence is grammatically acceptable.
Based on CoLA - Binary Grammatical Sentence acceptability classification

pipe = nlu.load('grammar_correctness')
# Set the task on T5
pipe['t5'].setTask('cola sentence: ')
# define Data
data = ['Anna and Mike is going skiing and they is liked is','Anna and Mike like to dance']
#Predict on text data with T5
pipe.predict(data)

|sentence | prediction| |————|————| | Anna and Mike is going skiing and they is liked is | unacceptable |
| Anna and Mike like to dance | acceptable |

Document Normalization

Document Normalizer example
The DocumentNormalizer extracts content from HTML or XML documents, applying either data cleansing using an arbitrary number of custom regular expressions either data extraction following the different parameters

pipe = nlu.load('norm_document')
data = '<!DOCTYPE html> <html> <head> <title>Example</title> </head> <body> <p>This is an example of a simple HTML page with one paragraph.</p> </body> </html>'
df = pipe.predict(data,output_level='document')
df

|text|normalized_text| |——|————-| | <!DOCTYPE html> <html> <head> <title>Example</title> </head> <body> <p>This is an example of a simple HTML page with one paragraph.</p> </body> </html> |Example This is an example of a simple HTML page with one paragraph.|

Word Segmenter

Word Segmenter Example
The WordSegmenter segments languages without any rule-based tokenization such as Chinese, Japanese, or Korean

pipe = nlu.load('ja.segment_words')
# japanese for 'Donald Trump and Angela Merkel dont share many opinions'
ja_data = ['ドナルド・トランプとアンゲラ・メルケルは多くの意見を共有していません']
df = pipe.predict(ja_data, output_level='token')
df

token
ドナルド
トランプ
アンゲラ
メルケル
多く
意見
共有
ませ

Installation

# PyPi
!pip install nlu pyspark==2.4.7
#Conda
# Install NLU from Anaconda/Conda
conda install -os_components johnsnowlabs nlu

Additional NLU ressources

NLU Version 1.0.6

Trainable Multi Label Classifiers, predict Stackoverflow Tags and much more in 1 Line of with NLU 1.0.6

We are glad to announce NLU 1.0.6 has been released! NLU 1.0.6 comes with the Multi Label classifier, it can learn to map strings to multiple labels. The Multi Label Classifier is using Bidirectional GRU and CNNs inside TensorFlow and supports up to 100 classes.

NLU 1.0.6 New Features

  • Multi Label Classifier
    • The Multi Label Classifier learns a 1 to many mapping between text and labels. This means it can predict multiple labels at the same time for a given input string. This is very helpful for tasks similar to content tag prediction (HashTags/RedditTags/YoutubeTags/Toxic/E2e etc..)
    • Support up to 100 classes
    • Pre-trained Multi Label Classifiers are already avaiable as Toxic and E2E classifiers

Multi Label Classifier

By default Universal Sentence Encoder Embeddings (USE) are used as sentence embeddings for training.

fitted_pipe = nlu.load('train.multi_classifier').fit(train_df)
preds = fitted_pipe.predict(train_df)

If you add a nlu sentence embeddings reference, before the train reference, NLU will use that Sentence embeddings instead of the default USE.

#Train on BERT sentence emebddings
fitted_pipe = nlu.load('embed_sentence.bert train.multi_classifier').fit(train_df)
preds = fitted_pipe.predict(train_df)

Configure a custom line seperator

#Use ; as label seperator
fitted_pipe = nlu.load('embed_sentence.electra train.multi_classifier').fit(train_df, label_seperator=';')
preds = fitted_pipe.predict(train_df)

NLU 1.0.6 Enhancements

  • Improved outputs for Toxic and E2E Classifier.
    • by default, all predicted classes and their confidences which are above the threshold will be returned inside of a list in the Pandas dataframe
    • by configuring meta=True, the confidences for all classes will be returned.

NLU Version 1.0.6

NLU 1.0.6 Bug-fixes

  • Fixed a bug that caused en.ner.dl.bert to be inaccessible
  • Fixed a bug that caused pt.ner.large to be inaccessible
  • Fixed a bug that caused USE embeddings not properly beeing configured to document level output when using multiple embeddings at the same time

NLU Version 1.0.5

Trainable Part of Speech Tagger (POS), Sentiment Classifier with BERT/USE/ELECTRA sentence embeddings in 1 Line of code! Latest NLU Release 1.0.5

We are glad to announce NLU 1.0.5 has been released!
This release comes with a trainable Sentiment classifier and a Trainable Part of Speech (POS) models!
These Neural Network Architectures achieve the state of the art (SOTA) on most binary Sentiment analysis and Part of Speech Tagging tasks!
You can train the Sentiment Model on any of the 100+ Sentence Embeddings which include BERT, ELECTRA, USE, Multi Lingual BERT Sentence Embeddings and many more!
Leverage this and achieve the state of the art in any of your datasets, all of this in just 1 line of Python code

NLU 1.0.5 New Features

  • Trainable Sentiment DL classifier
  • Trainable POS

NLU 1.0.5 New Notebooks and Tutorials

Sentiment Classifier Training

Sentiment Classification Training Demo

To train the Binary Sentiment classifier model, you must pass a dataframe with a ‘text’ column and a ‘y’ column for the label.

By default Universal Sentence Encoder Embeddings (USE) are used as sentence embeddings.

fitted_pipe = nlu.load('train.sentiment').fit(train_df)
preds = fitted_pipe.predict(train_df)

If you add a nlu sentence embeddings reference, before the train reference, NLU will use that Sentence embeddings instead of the default USE.

#Train Classifier on BERT sentence embeddings
fitted_pipe = nlu.load('embed_sentence.bert train.classifier').fit(train_df)
preds = fitted_pipe.predict(train_df)
#Train Classifier on ELECTRA sentence embeddings
fitted_pipe = nlu.load('embed_sentence.electra train.classifier').fit(train_df)
preds = fitted_pipe.predict(train_df)

Part Of Speech Tagger Training

Part Of Speech Tagger Training demo

fitted_pipe = nlu.load('train.pos').fit(train_df)
preds = fitted_pipe.predict(train_df)

NLU 1.0.5 Installation changes

Starting from version 1.0.5 NLU will not automatically install pyspark for users anymore.
This enables easier customizing the Pyspark version which makes it easier to use in various cluster enviroments.

To install NLU from now on, please run

pip install nlu pyspark==2.4.7 

or install any pyspark>=2.4.0 with pyspark<3

NLU 1.0.5 Improvements

  • Improved Databricks path handling for loading and storing models.

NLU Version 1.0.4

John Snow Labs NLU 1.0.4 : Trainable Named Entity Recognizer (NER) , achieve SOTA in 1 line of code and easy scaling to 100’s of Spark nodes

We are glad to announce NLU 1.0.4 releases the State of the Art breaking Neural Network architecture for NER, Char CNNs - BiLSTM - CRF!

#fit and predict in 1 line!
nlu.load('train.ner').fit(dataset).predict(dataset)


#fit and predict in 1 line with BERT!
nlu.load('bert train.ner').fit(dataset).predict(dataset)


#fit and predict in 1 line with ALBERT!
nlu.load('albert train.ner').fit(dataset).predict(dataset)


#fit and predict in 1 line with ELMO!
nlu.load('elmo train.ner').fit(dataset).predict(dataset)

Any NLU pipeline stored can now be loaded as pyspark ML pipeline

# Ready for big Data with Spark distributed computing
import pyspark
nlu_pipe.save(path)
pyspark_pipe = pyspark.ml.PipelineModel.load(stored_model_path)
pyspark_pipe.transform(spark_df)

NLU 1.0.4 New Features

NLU 1.0.4 New Notebooks,Tutorials and Docs

NLU 1.0.4 Bug Fixes

  • Fixed a bug that NER token confidences do not appear. They now appear when nlu.load(‘ner’).predict(df, meta=True) is called.
  • Fixed a bug that caused some Spark NLP models to not be loaded properly in offline mode

NLU Version 1.0.3

We are happy to announce NLU 1.0.3 comes with a lot new features, training classifiers, saving them and loading them offline, enabling running NLU with no internet connection, new notebooks and articles!

NLU 1.0.3 New Features

  • Train a Deep Learning classifier in 1 line! The popular ClassifierDL which can achieve state of the art results on any multi class text classification problem is now trainable! All it takes is just nlu.load(‘train.classifier).fit(dataset) . Your dataset can be a Pandas/Spark/Modin/Ray/Dask dataframe and needs to have a column named x for text data and a column named y for labels
  • Saving pipelines to HDD is now possible with nlu.save(path)
  • Loading pipelines from disk now possible with nlu.load(path=path).
  • NLU offline mode: Loading from disk makes running NLU offline now possible, since you can load pipelines/models from your local hard drive instead of John Snow Labs AWS servers.

NLU 1.0.3 New Notebooks and Tutorials

NLU 1.0.3 Bug fixes

  • Sentence Detector bugfix

NLU Version 1.0.2

We are glad to announce nlu 1.0.2 is released!

NLU 1.0.2 Enhancements

  • More semantically concise output levels sentence and document enforced :
    • If a pipe is set to output_level=’document’ :
      • Every Sentence Embedding will generate 1 Embedding per Document/row in the input Dataframe, instead of 1 embedding per sentence.
      • Every Classifier will classify an entire Document/row
      • Each row in the output DF is a 1 to 1 mapping of the original input DF. 1 to 1 mapping from input to output.
    • If a pipe is set to output_level=’sentence’ :
      • Every Sentence Embedding will generate 1 Embedding per Sentence,
      • Every Classifier will classify exactly one sentence
      • Each row in the output DF can is mapped to one row in the input DF, but one row in the input DF can have multiple corresponding rows in the output DF. 1 to N mapping from input to output.
  • Improved generation of column names for classifiers. based on input nlu reference
  • Improved generation of column names for embeddings, based on input nlu reference
  • Improved automatic output level inference
  • Various test updates
  • Integration of CI pipeline with Github Actions

New Documentation is out!

Check it out here : https://nlu.johnsnowlabs.com/

NLU Version 1.0.1

NLU 1.0.1 Bugfixes

  • Fixed bug that caused NER pipelines to crash in NLU when input string caused the NER model to predict without additional metadata

NLU Version 1.0.0

  • Automatic to Numpy conversion of embeddings
  • Added various testing classes
  • New 6 embeddings at once notebook with t-SNE and Medium article
  • Integration of Spark NLP 2.6.2 enhancements and bugfixes https://github.com/JohnSnowLabs/spark-nlp/releases/tag/2.6.2
  • Updated old T-SNE notebooks with more elegant and simpler generation of t-SNE embeddings

NLU Version 0.2.1

  • Various bugfixes
  • Improved output column names when using multiple classifirs at once

NLU Version 0.2.0

  • Improved output column names classifiers

NLU Version 0.1.0

We are glad to announce that NLU 0.1 has been released! NLU makes the 350+ models and annotators in Spark NLPs arsenal available in just 1 line of python code and it works with Pandas dataframes! A picture says more than a 1000 words, so here is a demo clip of the 12 coolest features in NLU, all just in 1 line!

NLU in action

What does NLU 0.1 include?

  • NLU provides everything a data scientist might want to wish for in one line of code!
  • 350 + pre-trained models
  • 100+ of the latest NLP word embeddings ( BERT, ELMO, ALBERT, XLNET, GLOVE, BIOBERT, ELECTRA, COVIDBERT) and different variations of them
  • 50+ of the latest NLP sentence embeddings ( BERT, ELECTRA, USE) and different variations of them
  • 50+ Classifiers (NER, POS, Emotion, Sarcasm, Questions, Spam)
  • 40+ Supported Languages
  • Labeled and Unlabeled Dependency parsing
  • Various Text Cleaning and Pre-Processing methods like Stemming, Lemmatizing, Normalizing, Filtering, Cleaning pipelines and more

NLU 0.1 Features Google Collab Notebook Demos

Last updated