Text input models are a type of machine learning model that operates on text data, such as natural language text found in documents, emails, social media posts, and other forms of written communication. These models are designed to learn the patterns and relationships between words and phrases in order to perform tasks such as sentiment analysis, language translation, and text classification. Text input models can be built using a variety of techniques, including neural networks, decision trees, and support vector machines.

Embeddings in Arthur

Text models in Arthur take in Text input, i.e. raw text of the documents or social media posts teams are using to predict. While there are a few enrichments (namely Anomaly Detection) that use model embeddings, Arthur computes these embeddings internally. Currently, Arthur does not take in embeddings or vector inputs.

Tokenization in Arthur

Text inputs are tokenized in Arthur for both anomaly detection and explainability. To create these tokens, Arthur has a few different text delimiters.

Text Delimiters

Here are the different delimiters available in Arthur.