Inference on sub-topics

[DobromirM]

After creating a hierarchical model by subdividing the top groups into sub-groups, how can we infer topical content for new documents with the fitted model?

The example shown in the documentation returns only the inference for the top-level categories.

document_topic_matrix = model.transform(new_documents, embeddings=None)

I.e. if we start with a model with 5 categories and then sub-divide each category into 3 sub-categories, the transform() method returns the scores only for the top 5 categories instead of the scores for all 5 top categories + 15 sub-categories.

[x-tabdeveloping]

Hi @DobromirM ! I have yet to think of a way to do this, and this functionality is not implemented in the library yet, but I have a few ideas for it. I'm on holidays right now but I can take a look at it in a couple of weeks. If you were to have a go at it yourself I'd really appreciate it.

[DobromirM]

@x-tabdeveloping I've been working on an implementation and will open a PR once ready!

[x-tabdeveloping]

That's awesome! Let me know if youneed help or code review

[DobromirM]

I did the following test implementation which uses the 20newsgroups data.

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer
import warnings
import numpy as np
from sklearn.decomposition import NMF
from sklearn.exceptions import NotFittedError
from turftopic import KeyNMF
from sklearn.utils.validation import check_non_negative
from turftopic.models.wnmf import weighted_nmf


# Helper method to get the document topic metrix for our example
def get_document_topic_matrix(topic_data):
    document_topic_matrix = np.column_stack(
        (topic_data.hierarchy[0].document_topic_vector, topic_data.hierarchy[1].document_topic_vector,
         topic_data.hierarchy[2].document_topic_vector, topic_data.hierarchy[3].document_topic_vector,
         topic_data.hierarchy[4].document_topic_vector, topic_data.hierarchy[0][0].document_topic_vector,
         topic_data.hierarchy[0][1].document_topic_vector, topic_data.hierarchy[0][2].document_topic_vector))

    return document_topic_matrix


# Test function that tries to infer the subgroups
def infer(model, raw_documents=None, embeddings=None, keywords=None):
    if model.seed_phrase is not None and keywords is not None:
        warnings.warn(
            "Seed phrase is specified, but keyword matrix is pre-computed. The seed phrase will be ignored. Note that this is not a problem if you already calculated the keyword matrix using the seed phrase."
        )
    if keywords is None and raw_documents is None:
        raise ValueError(
            "You have to pass either keywords or raw_documents."
        )
    if keywords is None:
        keywords = model.extract_keywords(
            list(raw_documents),
            embeddings=embeddings,
            fitting=False,
        )

    internal_model = model.model

    if internal_model.components is None:
        raise NotFittedError(
            "Can't transform() if the model has not been fitted."
        )
    X = internal_model.vectorize(keywords, fitting=False)
    check_non_negative(X, "NMF (input X)")

    hierarchy_components = model.hierarchy.components_

    W, _, _ = NMF(
        len(hierarchy_components), init="custom", random_state=internal_model.seed
    )._fit_transform(X, W=None, H=hierarchy_components, update_H=False)

    document_topic_matrix = W.astype(X.dtype)

    subcomponents, sub_doc_topic = weighted_nmf(
        X, document_topic_matrix[:, 0], len(model.hierarchy[0].components_),
        internal_model.seed, max_iter=200
    )

    document_topic_matrix = np.concatenate([document_topic_matrix, sub_doc_topic], axis=1)

    return document_topic_matrix


if __name__ == '__main__':
    corpus = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes')).data
    corpus = corpus[0:250]
    vectorizer = CountVectorizer(min_df=5, max_df=0.8, stop_words='english')

    model = KeyNMF(n_components=5, vectorizer=vectorizer)
    topic_data = model.prepare_topic_data(corpus)
    topic_data.hierarchy[0].divide(3)

    document_topic_matrix_original = get_document_topic_matrix(topic_data)

    # These are inferred as the original data for all groups and subgroups
    document_topic_matrix_inferred = infer(model, corpus)

    # These are inferred as the original data for the top level groups but the inference for the subgroups is different
    document_topic_matrix_inferred = infer(model, corpus[0:25])

I am using this as a sanity check to confirm that the inference method returns the same results as the training method when given the same documents.

The test implementation infers the same groups for the documents when the full set of documents is passed, however if only a part of the documents is passed, the inference of the subgroups diverges.

[DobromirM]

I think the problem is that weighted_nmf re-calculates the sub-components, but I am not sure how to use the already existing components on the child nodes with the weighted_nmf in order to get the same document_topic_matrix as the training procedure.

[DobromirM]

The alternative implementation that I tested was using NMF directly with the computed sub-components like this:

def infer(model, raw_documents=None, embeddings=None, keywords=None):
    if model.seed_phrase is not None and keywords is not None:
        warnings.warn(
            "Seed phrase is specified, but keyword matrix is pre-computed. The seed phrase will be ignored. Note that this is not a problem if you already calculated the keyword matrix using the seed phrase."
        )
    if keywords is None and raw_documents is None:
        raise ValueError(
            "You have to pass either keywords or raw_documents."
        )
    if keywords is None:
        keywords = model.extract_keywords(
            list(raw_documents),
            embeddings=embeddings,
            fitting=False,
        )

    internal_model = model.model

    if internal_model.components is None:
        raise NotFittedError(
            "Can't transform() if the model has not been fitted."
        )
    X = internal_model.vectorize(keywords, fitting=False)
    check_non_negative(X, "NMF (input X)")

    hierarchy_components = model.hierarchy.components_

    W, _, _ = NMF(
        len(hierarchy_components), init="custom", random_state=internal_model.seed
    )._fit_transform(X, W=None, H=hierarchy_components, update_H=False)

    document_topic_matrix = W.astype(X.dtype)

    W, _, _ = NMF(
        len(model.hierarchy[0].components_), init="custom", random_state=internal_model.seed
    )._fit_transform(X, W=None, H=model.hierarchy[0].components_, update_H=False)

    document_topic_matrix = np.concatenate([document_topic_matrix, W.astype(X.dtype)], axis=1)

    return document_topic_matrix

However, that results in different results compared to the training method for both

    document_topic_matrix_inferred = infer(model, corpus)

    document_topic_matrix_inferred = infer(model, corpus[0:25])