In the user experience (UX) research teams of large companies like Telefónica, we need to do a lot of document management before starting a new research. We do market research on what other companies have published on a topic, to find out what they have explored and what are the main UX research conclusions they have drawn. On many occasions we research topics that are enriched by what others have learned.
Searching for all this information can be tedious, because this content is often scattered across multiple repositories and the information is often not well labeled, nor do the titles reflect the content of the documents. For this reason, researching requires conscientious reading of more and more documents, which entails an investment of time and effort that we often do not have.
Shared repositories within an organization solve only the access part, but not the location part. The larger the repository, the more complicated it is to find the content you are looking for.
Artificial intelligence (AI) applied to UX research
Little by little, user experience research has evolved to make it faster to find specific information, but more traditional search interfaces present difficulties, such as requiring specialized knowledge or remaining highly dependent on tagging.
Therefore, generative artificial intelligence (AI), applied to UX, can be of great help to solve these difficulties, offering an intuitive interface (questions and queries in natural language) and conversational (allowing a “dialogue with the library”), so that all the knowledge stored in the repository can be put to good use.
One of the advantages offered by technology when integrating AI is that it provides an answer to the eternal debate as to whether or not the content has been generated by humans. With applications such as VerifAI, it is now possible to detect whether content has been manipulated or created by artificial intelligence.
In Telefónica’s Discovery team we have worked on a system based on RAG (Retrieval-Augmented Generation) architecture, used with modern generative AI techniques, and which has three differentiated elements:
- Extraction of documentation from the document repository: as we are going to use a language model, what we extract will be the text contained in the documents, trying to maintain as much as possible its structure.This extracted text is segmented into smaller portions to be inserted into a data base that performs a double search: both by semantic similarity and by coincidence of terms to give more precision.In addition, the search system does not consider all terms in the same way, but the most identifying ones for a particular content are more important in the searches. This form of data storage makes it possible to simultaneously handle large amounts of information, as well as to facilitate specific searches.
- To extract value from the stored content, the experimenters’ requests are converted into searches (semantic and term-based) in the previous database. The most meaningful results with respect to the experimenter’s request are obtained. This step solves the problem of an open query, since it covers both more concrete terms and related content, even if different terms than the original ones are used.
- A large-scale language model (LLM) uses all the retrieved information, the experimenter’s request and the context of the conversation to produce a clear and appropriate response in natural language, solving the problem of the difficulty of interacting with the information without specialized knowledge.
To allow more research possibilities for users, all results include the textual response and references (as detailed as possible) to the sources of the information in the collected documents, so that users can analyze them and contrast the original texts.
Subscribe to our newsletter!
Find out about our offers and news before anyone else
UX research, a project in development
The tests carried out have allowed us to demonstrate the feasibility and potential usefulness of the system. This research is still in progress and has a twofold objective in the medium term:
- On the one hand, to carry out more extensive tests, integrating more documents and offering the use of the system to UX research teams so that they can use it in their day-to-day work. To this end, the environment contains an element that allows the user to provide immediate feedback on the suitability of the results; by collecting this signal we can analyze failures and successes to see where to have an impact.
- On the other hand, the initial evaluation has already identified a number of possible improvements, which we are working on. For example, the conversational capability (rated as one of the most useful features since it allows us to refine a question/answer) is sometimes hampered by the problems the models have in recognizing when the user is changing context. Therefore, a mechanism for identifying such changes has been introduced to produce smoother results.
Overall, this is a first step in creating intelligent systems, but also a first step that is already useful and practical for our internal research teams.
Key findings of the research
The system has been deployed as a proof of concept, integrating a set of documents and reports of past research results from Telefónica’s different UX groups. A working session was held with 8 research teams in which they were presented with the system and were able to interact with it collecting impressions, opinions and proposals for improvement to facilitate their professional practice.
The main conclusions were positive. The teams recognized that the value of having such a tool lay in:
- Reduction of time in the search for documentation.
- Increased quality of analysis.
- Improved collaboration between research teams and the possibility of cross-referencing.
- Improvement of the creative process in the selection of research techniques.
They especially highlighted the ability to collect information from documents in multiple formats (different languages, different information structures, graphics, etc.) and identified a series of use cases and improvements to implement in the system to improve its efficiency, such as increasing the amount of referential information per document.
On the other hand, they perceived a risk associated with the misuse of the system by non-experienced profiles that would draw conclusions based on their own conversation without verifying and contrasting the references and documents provided in the results. This could lead to erroneous conclusions and wrong decisions, which underlines the importance of proper training and knowledge to use the system effectively.
This post has been made in collaboration with Ana Mendiola and Paulo Villegas from the Discovery area of Telefónica Innovación Digital.
