Quoting source from given text for Q&A prompt
I'm experimenting with using Flan-T5 for question answering from a given chunk of text. It is pretty reliable about giving decent answers but I would like to it cite the text from which it got its answer in an exact quote, so that I can highlight the text in a user interface, and also because LLMs can lie. I tried prepping the model with some few shot examples quoting the text in the answer but this did not change the behavior at all. I've had some limited success with this by adding "How do you know that?" or "Where did you learn that?" or "Where does it say that?" to the end of my question. I get slightly different results with each approach, and unfortunately sometimes adding this term reduces the accuracy of the response, or also sometimes it quotes text that is nearby the answer but that's not correct. I'm wondering if I'm confusing it or using the wrong approach. Is there a best practice for getting Flan-T5 to produce a quote or quotes of the text from the given source? Is there a better place for me to ask this question, that's more general to FLAN?
I'm using this for mining a large dataset of transcribed interviews and narratives for evidence of people doing certain things. LLMs seems to work well for this task because there can be a lot of linguistic nuance, which traditional NLP methods can't cope with to the same degree.
Thanks!
sorry, I want to use flan_t5 for questions and answers. I have challenges and for each challenge the related solution and no more contex. would you please guide me that how I can create an appropriate prompt for that? Should be in one column and include both question and answer or 2 different columns? I generate two different columns like "answer the following question: what is the solution for ...., another column as the solution is.....
can I use seq2seq or T5conditionalgeneration and Causal type? many thanks
Hi @SUNM your question seems different from the topic of this thread, which is about quoting text from the given context. Please start a new thread for your specific question so that other people can see it and reply to it without confusing this one. Thanks!
Since this conversation has been quiet, I'll share what else I have learned on my own about this. Lately I've been having success using the phrase "quote the text that proves it". This has been working pretty reliably for me. However, when compounded with other directives in a single prompt the model can get confused. E.g. "Was someone at the store at 3pm? Quote the text that proves it. Say unknown if you don't know." Just overwhelms the model and the quality of the responses is very poor, especially when an answer can't actually be found in the given text. I have resorted to a multistep sequence of prompts, first with the bare question "Was someone at the store at 3pm? Say unknown if you don't know.", then if I get a good quality answer I reprompt including the clause "Quote the text that proves it." This seems to work pretty well for my situation, but it's far from ideal.