LLMs for Feature Engineering

It’s 2025 and Large Language Models (LLMs) are still all the rage. They’ve been shown to be useful at many tasks, notably classification information extraction, although most people seem interested in their generative applications. When it comes to LLMs my professional interest is primarily in their feature embeddings.

Embeddings are a vector representation produced by the LLM. These vectors can be computed once and stored in a vector database for later use. The vectors can be used for tasks like nearest neighbor (similarity) search and recommendation engines. The embeddings can also be used as input to traditional machine learning tasks.

Feature embeddings can be used with unsupervised learning models to produce clusters of similar data. This can be very helpful when exploring new datasets and to help uncover how the data is organized in latent space.

They can also be used in supervised learning models. While generative pretrained and transformers can be used to perform classification, in my experience these models rarely outperform classical machine learning methods. Instead I often take the feature embeddings and using them in a classifier like gradient boosting trees or support vector machines.