The Power of GenAI: Transform Unstructured Data into Valuable Insights
As a data specialist or data scientist, you've undoubtedly encountered the challenge of unstructured and nominal data. Many variables cannot be included because the data is unstructured or only at a nominal measurement level. This limits not only the effectiveness of your models but also the depth of your analyses.
As a data specialist or data scientist, you've undoubtedly encountered the challenge of unstructured and nominal data. You're busy building a predictive AI model or performing a regression analysis, but you're running into limitations. Many variables cannot be included because the data is unstructured or only at a nominal measurement level. This limits not only the effectiveness of your models but also the depth of your analyses.
But what if there's a way to break through these barriers? In this article, we delve deep into how Generative AI (GenAI), and specifically language models, can be used to transform unstructured data into structured, analysable information. This allows you to include more variables in your models, increase the proportion of explained variance, and significantly improve the quality of your predictive analyses.
The Problem with Unstructured and Nominal Data
Unstructured Data: This refers to data that doesn't fit into traditional database formats. Think of texts, images, audio, and video. They lack a predefined data structure, making them difficult to analyse with conventional methods.
Nominal Data: This refers to categorical data without intrinsic order or numerical value. Examples include gender, colour, or brand preference. They offer limited possibilities for statistical analyses because there is no numerical variation.
Limitations in Models:
- Non-numeric: Without numerical values, many algorithms cannot process the data.
- Limited Variability: Nominal data lack the scale and variation needed for in-depth analyses.
- Complexity: Manually structuring this data is time-consuming and often impractical.
The Rise of Generative AI (GenAI)
GenAI, and especially advanced language models, have radically changed the way we approach data. These models are capable of understanding, generating, and interpreting human language at a level previously unattainable.
What is GenAI?
Generative AI refers to algorithms capable of creating new content, ranging from text and images to music and more. Language models like GPT-4 are trained on massive datasets and can understand and reproduce context.
Why Language Models?
- Natural Language Processing (NLP): They can analyse and interpret unstructured texts.
- Contextual Understanding: They understand the nuances and meanings behind words and sentences.
- Automation: They can process large amounts of data quickly without human intervention.
Transformation from Unstructured to Structured Data
How Does It Work?
- Data Input: Feed the unstructured data into the language model.
- Training and Fine-Tuning: Adapt the model to specific domains or terminologies for accuracy.
- Information Extraction: The model identifies relevant information and categorises it.
- Structuring: The extracted data is converted into structured formats, such as tables or numerical values.
Examples of Applications:
- Sentiment Analysis: Determining the emotional tone in customer feedback.
- Entity Extraction: Identifying names, locations, and other key terms from texts.
- Classification: Categorising documents based on content.
From Nominal to Interval Measurement Level
One of the greatest advantages of using GenAI is the ability to convert nominal data to data of a higher measurement level, such as interval data.
What Does This Mean?
- Nominal Measurement Level: Categorical data without order (e.g., blood type).
- Interval Measurement Level: Numerical data with equal intervals between values, but without an absolute zero point (e.g., temperature in Celsius).
How GenAI Helps:
- Scoring and Assessment: Language models can score sentiments or characteristics on a scale.
- Determining Intensity: They can quantify the intensity of certain features.
- Comparability: This makes the data suitable for statistical analyses that offer more in-depth insights.
Benefits for Predictive Models and Regression Analyses
More Variables, Better Models
By transforming unstructured and nominal data, you can:
- Increased Explained Variance: More variables lead to a higher R-squared in regression analyses.
- Better Predictions: Models become more accurate with more relevant input data.
- Deeper Insights: Discover hidden patterns and relationships that were previously invisible.
Efficiency and Scalability
- Automation: Save time and resources by automating processes.
- Consistency: Reduce human errors and improve data quality.
- Scalability: Process huge amounts of data without additional overhead.
Practical Steps to Implement GenAI
- Identify Unstructured Data Sources: Text files, emails, PDFs, social media posts, etc.
- Choose the Right Language Model: Use existing models or train your own model with specific data.
- Training and Validation: Ensure a representative dataset for training. Validate the output to ensure accuracy.
- Integration into Existing Workflows: Link the output of the language model to your data analysis tools.
- Monitor and Optimise: Track performance and adjust the model as needed.
- Text files, emails, PDFs, social media posts, etc.
- Use existing models or train your own model with specific data.
- Ensure a representative dataset for training.
- Validate the output to ensure accuracy.
- Link the output of the language model to your data analysis tools.
- Track performance and adjust the model as needed.
The integration of GenAI and language models into data analysis offers unprecedented opportunities. By transforming unstructured and nominal data into structured, numerical formats, you can overcome the limitations of traditional models. This leads to richer insights, more accurate predictions, and a significant improvement in the quality of your AI models.
I call upon all data specialists and data scientists to embrace this technology. Experiment with GenAI, integrate it into your workflows, and discover the benefits for yourself. The future of data analysis lies in the ability to utilise all available information – and GenAI is the key to that potential.
- Best AI Tools
- GEO: discoverable in AI search engines
- Writing AI prompts
Job van den Berg is an AI keynote speaker, tech entrepreneur and author of five books on AI. He ships AI agents into production every week and delivers 150+ keynotes a year on AI agents and agentic commerce.
On EditieNL I discussed whether AI could threaten humanity. About Anthropic researcher Evan Hubinger, agentic AI, the black box, and why we are building faster than we understand.
A simple AI video already costs about 4 litres of water and as much electricity as a 10-watt LED lamp burning for 42 hours. What happens with full films and commercials, and why digital is not automatically sustainable.
AI keeps getting better, yet workplace sentiment about AI is deteriorating. Research shows why adoption is as much a social challenge as a technological one: from the Matthew effect to psychological safety.
Human in the loop sounds reassuring. But researchers warn that prolonged use of autonomous AI systems can undermine the cognitive capacities of the very supervisors we depend on. The question is not whether a human is formally present, but whether that human can still meaningfully intervene.