| \n"," | Q | \n","A | \n","
|---|---|---|
| 0 | \n","What does article 4 say regarding students who... | \n","Article 4 states that such students will still... | \n","
| 1 | \n","How many days after publication would it take ... | \n","This law becomes effective thirty days from it... | \n","
| 2 | \n","In which year did President Luiz Inacio Lula D... | \n","Law 12.089 was signed by him in November of 2009 | \n","
| 3 | \n","Is there an exception allowing someone current... | \n","Yes | \n","
| 4 | \n","Can you enroll in only a single vacancy even i... | \n","No | \n","
| ... | \n","... | \n","... | \n","
| 1032 | \n","Is there flexibility allowed in terms of proje... | \n","No specific details were mentioned explicitly ... | \n","
| 1033 | \n","Are qualifications required for teaching staff... | \n","Academic qualification is among factors evalua... | \n","
| 1034 | \n","Does infrastructure play a role in determining... | \n","Infrastructure facilities form part of the par... | \n","
| 1035 | \n","Will financial resources impact an institute's... | \n","Financial capacity forms part of consideration... | \n","
| 1036 | \n","Do students who complete recognized degree pro... | \n","This aspect has been left unclear, but presuma... | \n","
1037 rows × 2 columns
\n","| \n"," | prompt | \n","response | \n","
|---|---|---|
| 0 | \n","What does article 4 say regarding students who... | \n","Article 4 states that such students will still... | \n","
| 1 | \n","How many days after publication would it take ... | \n","This law becomes effective thirty days from it... | \n","
| 2 | \n","In which year did President Luiz Inacio Lula D... | \n","Law 12.089 was signed by him in November of 2009 | \n","
| 3 | \n","Is there an exception allowing someone current... | \n","Yes | \n","
| 4 | \n","Can you enroll in only a single vacancy even i... | \n","No | \n","
| ... | \n","... | \n","... | \n","
| 1032 | \n","Is there flexibility allowed in terms of proje... | \n","No specific details were mentioned explicitly ... | \n","
| 1033 | \n","Are qualifications required for teaching staff... | \n","Academic qualification is among factors evalua... | \n","
| 1034 | \n","Does infrastructure play a role in determining... | \n","Infrastructure facilities form part of the par... | \n","
| 1035 | \n","Will financial resources impact an institute's... | \n","Financial capacity forms part of consideration... | \n","
| 1036 | \n","Do students who complete recognized degree pro... | \n","This aspect has been left unclear, but presuma... | \n","
1008 rows × 2 columns
\n","| Step | \n","Training Loss | \n","
|---|---|
| 25 | \n","3.306000 | \n","
| 50 | \n","2.150400 | \n","
| 75 | \n","1.889600 | \n","
| 100 | \n","1.566900 | \n","
| 125 | \n","1.763500 | \n","
| 150 | \n","1.533900 | \n","
| 175 | \n","1.708100 | \n","
| 200 | \n","1.494000 | \n","
| 225 | \n","1.658700 | \n","
| 250 | \n","1.696700 | \n","
| 275 | \n","1.471800 | \n","
| 300 | \n","1.568200 | \n","
| 325 | \n","1.449200 | \n","
| 350 | \n","1.629500 | \n","
| 375 | \n","1.411000 | \n","
| 400 | \n","1.632000 | \n","
| 425 | \n","1.473900 | \n","
| 450 | \n","1.563500 | \n","
| 475 | \n","1.596500 | \n","
| 500 | \n","1.383300 | \n","
| 525 | \n","1.538500 | \n","
| 550 | \n","1.382700 | \n","
| 575 | \n","1.524500 | \n","
| 600 | \n","1.422500 | \n","
| 625 | \n","1.499200 | \n","
| 650 | \n","1.473400 | \n","
| 675 | \n","1.467500 | \n","
"]},"metadata":{}},{"output_type":"execute_result","data":{"text/plain":["('TinyLlama-1.1B-Chat-v1.0/tokenizer_config.json',\n"," 'TinyLlama-1.1B-Chat-v1.0/special_tokens_map.json',\n"," 'TinyLlama-1.1B-Chat-v1.0/tokenizer.model',\n"," 'TinyLlama-1.1B-Chat-v1.0/added_tokens.json',\n"," 'TinyLlama-1.1B-Chat-v1.0/tokenizer.json')"]},"metadata":{},"execution_count":14}]},{"cell_type":"code","source":["torch.save(trainer.model.state_dict(), 'model.pt') # salva o modelo em pytorch"],"metadata":{"id":"fbVGW--ayojA"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### **Teste e Avaliação do modelo**\n","\n","Agora, vamos realizar os testes para conseguir avaliar o quão bem o modelo está conseguindo responder as questões para dados de teste (do dataframe `test_df`).\n","\n","Primeiro, configuramos a **verbosidade** do log para nível crítico, garantindo que apenas mensagens críticas sejam exibidas, o que ajuda a manter a saída do log limpa durante a execução dos testes. Em seguida, configuramos o **pipeline** para geração de texto, definindo os seguintes parâmetros:\n","- Tarefa: geração de texto\n","- Modelo: TinyLlama-1.1B-Chat-v1.0\n","- Tokenizador: AutoTokenizer\n","- Número máximo de novos tokens a serem gerados: 100 tokens"],"metadata":{"id":"YMFg-qgBrjkA"}},{"cell_type":"code","source":["logging.basicConfig(level=logging.CRITICAL)\n","\n","pipe = pipeline(task=\"text-generation\", model=model, tokenizer=tokenizer, max_new_tokens=100)"],"metadata":{"id":"FLc1p2qHpBGW"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["Abaixo definimos uma função chamada `obtem_resposta`, que gera uma resposta para uma pergunta fornecida, utilizando o Tiny Llama, e depois trunca a resposta na primeira quebra de linha. A razão desse truncamento é que o modelo estava gerando múltiplos pares de perguntas e respostas, o que deve ser em virtude a simplicidade do modelo. Por isso, truncamos para obter somente a primeira resposta, referente a pergunta passada para o modelo."],"metadata":{"id":"zOBjzp87r9TQ"}},{"cell_type":"code","source":["def obtem_resposta(question, cont):\n"," system_message = \"You are an assistant specialized in providing clear and precise answers on legal and administrative matters. Answer in a technical and detailed manner. Answer in English.\"\n"," prompt = f\"{system_message}\\n\\n{question}\"\n","\n"," result = pipe(prompt)\n"," print(f\"Gerei a resposta {cont} do Tiny..\")\n","\n"," generated_text = result[0]['generated_text']\n","\n"," prompt_end = prompt.split('\\n\\n')[-1]\n"," if prompt_end in generated_text:\n"," generated_text = generated_text.split(prompt_end, 1)[-1].strip()\n","\n"," truncated_response = generated_text.split('\\n')[0].strip() # truncar\n","\n"," return truncated_response\n","# exemplo de uso\n","# question = \"Quantos dias após a publicação, uma lei levaria para entrar em vigor de acordo com o Artigo 5?\"\n","# answer = obtem_resposta(question)\n","# print(answer)"],"metadata":{"id":"-_MyzXbepFXX"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["A função `avaliar_modelo` abaixo é responsável por avaliar o modelo no conjunto de teste aplicando a função acima, que obtém a resposta do modelo para a pergunta do teste. Ela itera sobre cada linha do dataframe de teste, gerando respostas para as perguntas fornecidas e comparando-as com as respostas esperadas. Os resultados são armazenados em uma lista, que é convertida em um DataFrame e salva em um arquivo CSV. Por fim, o DataFrame de resultados é impresso."],"metadata":{"id":"ymFmYbH6t4do"}},{"cell_type":"code","source":["def avaliar_modelo(test_df):\n"," cont = 1\n"," results = []\n","\n"," for i, row in test_df.iterrows():\n"," question = row['prompt']\n"," expected_answer = row['response']\n"," generated_answer = obtem_resposta(question, cont)\n"," cont += 1\n","\n"," results.append({'prompt': question,\n"," 'resposta_real': expected_answer,\n"," 'resposta_prevista': generated_answer})\n"," # print(f\"Pergunta: {question}\")\n"," # print(f\"Resposta Esperada: {expected_answer}\")\n"," # print(f\"Resposta Gerada: {generated_answer}\")\n"," # print(\"-\" * 50)\n"," # break\n","\n","\n"," results_df = pd.DataFrame(results)\n","\n"," results_df.to_csv('results.csv', index=False, sep='\\t')\n","\n"," print(results_df)\n"," return results_df"],"metadata":{"id":"jGXDvLHAuQXD"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["results_df = avalia_modelo(test_df)"],"metadata":{"id":"L2jVp3PBBeDB","colab":{"base_uri":"https://localhost:8080/"},"outputId":"ef542de0-6f48-4690-b26b-7a0fe125ea56"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Gerei a resposta 1 do Tiny..\n","Gerei a resposta 2 do Tiny..\n","Gerei a resposta 3 do Tiny..\n","Gerei a resposta 4 do Tiny..\n","Gerei a resposta 5 do Tiny..\n","Gerei a resposta 6 do Tiny..\n","Gerei a resposta 7 do Tiny..\n","Gerei a resposta 8 do Tiny..\n","Gerei a resposta 9 do Tiny..\n"]},{"output_type":"stream","name":"stderr","text":["You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset\n"]},{"output_type":"stream","name":"stdout","text":["Gerei a resposta 10 do Tiny..\n","Gerei a resposta 11 do Tiny..\n","Gerei a resposta 12 do Tiny..\n","Gerei a resposta 13 do Tiny..\n","Gerei a resposta 14 do Tiny..\n","Gerei a resposta 15 do Tiny..\n","Gerei a resposta 16 do Tiny..\n","Gerei a resposta 17 do Tiny..\n","Gerei a resposta 18 do Tiny..\n","Gerei a resposta 19 do Tiny..\n","Gerei a resposta 20 do Tiny..\n","Gerei a resposta 21 do Tiny..\n","Gerei a resposta 22 do Tiny..\n","Gerei a resposta 23 do Tiny..\n","Gerei a resposta 24 do Tiny..\n","Gerei a resposta 25 do Tiny..\n","Gerei a resposta 26 do Tiny..\n","Gerei a resposta 27 do Tiny..\n","Gerei a resposta 28 do Tiny..\n","Gerei a resposta 29 do Tiny..\n","Gerei a resposta 30 do Tiny..\n","Gerei a resposta 31 do Tiny..\n","Gerei a resposta 32 do Tiny..\n","Gerei a resposta 33 do Tiny..\n","Gerei a resposta 34 do Tiny..\n","Gerei a resposta 35 do Tiny..\n","Gerei a resposta 36 do Tiny..\n","Gerei a resposta 37 do Tiny..\n","Gerei a resposta 38 do Tiny..\n","Gerei a resposta 39 do Tiny..\n","Gerei a resposta 40 do Tiny..\n","Gerei a resposta 41 do Tiny..\n","Gerei a resposta 42 do Tiny..\n","Gerei a resposta 43 do Tiny..\n","Gerei a resposta 44 do Tiny..\n","Gerei a resposta 45 do Tiny..\n","Gerei a resposta 46 do Tiny..\n","Gerei a resposta 47 do Tiny..\n","Gerei a resposta 48 do Tiny..\n","Gerei a resposta 49 do Tiny..\n","Gerei a resposta 50 do Tiny..\n","Gerei a resposta 51 do Tiny..\n","Gerei a resposta 52 do Tiny..\n","Gerei a resposta 53 do Tiny..\n","Gerei a resposta 54 do Tiny..\n","Gerei a resposta 55 do Tiny..\n","Gerei a resposta 56 do Tiny..\n","Gerei a resposta 57 do Tiny..\n","Gerei a resposta 58 do Tiny..\n","Gerei a resposta 59 do Tiny..\n","Gerei a resposta 60 do Tiny..\n","Gerei a resposta 61 do Tiny..\n","Gerei a resposta 62 do Tiny..\n","Gerei a resposta 63 do Tiny..\n","Gerei a resposta 64 do Tiny..\n","Gerei a resposta 65 do Tiny..\n","Gerei a resposta 66 do Tiny..\n","Gerei a resposta 67 do Tiny..\n","Gerei a resposta 68 do Tiny..\n","Gerei a resposta 69 do Tiny..\n","Gerei a resposta 70 do Tiny..\n","Gerei a resposta 71 do Tiny..\n","Gerei a resposta 72 do Tiny..\n","Gerei a resposta 73 do Tiny..\n","Gerei a resposta 74 do Tiny..\n","Gerei a resposta 75 do Tiny..\n","Gerei a resposta 76 do Tiny..\n","Gerei a resposta 77 do Tiny..\n","Gerei a resposta 78 do Tiny..\n","Gerei a resposta 79 do Tiny..\n","Gerei a resposta 80 do Tiny..\n","Gerei a resposta 81 do Tiny..\n","Gerei a resposta 82 do Tiny..\n","Gerei a resposta 83 do Tiny..\n","Gerei a resposta 84 do Tiny..\n","Gerei a resposta 85 do Tiny..\n","Gerei a resposta 86 do Tiny..\n","Gerei a resposta 87 do Tiny..\n","Gerei a resposta 88 do Tiny..\n","Gerei a resposta 89 do Tiny..\n","Gerei a resposta 90 do Tiny..\n","Gerei a resposta 91 do Tiny..\n","Gerei a resposta 92 do Tiny..\n","Gerei a resposta 93 do Tiny..\n","Gerei a resposta 94 do Tiny..\n","Gerei a resposta 95 do Tiny..\n","Gerei a resposta 96 do Tiny..\n","Gerei a resposta 97 do Tiny..\n","Gerei a resposta 98 do Tiny..\n","Gerei a resposta 99 do Tiny..\n","Gerei a resposta 100 do Tiny..\n","Gerei a resposta 101 do Tiny..\n"," prompt \\\n","0 How many days after publication would it take ... \n","1 May i know whom publicly issued interpertation... \n","2 Does the act specify who has authority over gr... \n","3 Are field assignments subject to application f... \n","4 Can a university issue a degree with specific ... \n",".. ... \n","96 When did Resolution Nº 027/2010-CEG come into ... \n","97 In what circumstances would a student receive ... \n","98 Are only Brazilian institutions considered eli... \n","99 Will students automatically receive full credi... \n","100 Where will requests for accreditation from ins... \n","\n"," resposta_real \\\n","0 This law becomes effective thirty days from it... \n","1 The listed government officials authored rule ... \n","2 Granting falls upon coordinators' duties with ... \n","3 Field works aren't explicitly mentioned being ... \n","4 No, they cannot. According to the response fro... \n",".. ... \n","96 Resolution Nº 027/2010-CEG came into effect on... \n","97 A student who registered but wasn't selected b... \n","98 Foreign higher education institutions may also... \n","99 Meeting the minimal percentage thresholds alon... \n","100 Requests for accreditation from institutions w... \n","\n"," resposta_prevista \n","0 Answer according to: Article 5: This law shall... \n","1 Sure, the publicly issued interpertation guida... \n","2 Answer according to… The act specifies that th... \n","3 Yes, field assignments are subject to applicat... \n","4 Yes, a university can issue a degree with spec... \n",".. ... \n","96 Resolution Nº 027/2010-CEG came into effect on... \n","97 A student receives a mention of 'dispensed' in... \n","98 No, any institution recognized by the Brazilia... \n","99 No, credit will only be awarded if the student... \n","100 In the Ministry of Education's office in the c... \n","\n","[101 rows x 3 columns]\n"]}]},{"cell_type":"code","source":["test_question = \"Whats is UFAM?\"\n","answer = obtem_resposta(test_question, 1)\n","print(answer)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"jZ-0I6qmiaPW","outputId":"5c840ac3-9271-49e2-ddcc-9ab2e0a9a753"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Gerei a resposta 1 do Tiny..\n","What is its mission? Provide a brief overview of its activities and objectives.\n"]}]},{"cell_type":"code","source":["# salvando no csv\n","results_df.to_csv('results_LORA.csv', index=False, sep='\\t')\n","\n","print(results_df)"],"metadata":{"id":"_VdzTF6lL0Fa","colab":{"base_uri":"https://localhost:8080/"},"outputId":"7ea890b5-ef92-4513-dfcf-78a065b69557"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":[" prompt \\\n","0 How many days after publication would it take ... \n","1 May i know whom publicly issued interpertation... \n","2 Does the act specify who has authority over gr... \n","3 Are field assignments subject to application f... \n","4 Can a university issue a degree with specific ... \n",".. ... \n","96 When did Resolution Nº 027/2010-CEG come into ... \n","97 In what circumstances would a student receive ... \n","98 Are only Brazilian institutions considered eli... \n","99 Will students automatically receive full credi... \n","100 Where will requests for accreditation from ins... \n","\n"," resposta_real \\\n","0 This law becomes effective thirty days from it... \n","1 The listed government officials authored rule ... \n","2 Granting falls upon coordinators' duties with ... \n","3 Field works aren't explicitly mentioned being ... \n","4 No, they cannot. According to the response fro... \n",".. ... \n","96 Resolution Nº 027/2010-CEG came into effect on... \n","97 A student who registered but wasn't selected b... \n","98 Foreign higher education institutions may also... \n","99 Meeting the minimal percentage thresholds alon... \n","100 Requests for accreditation from institutions w... \n","\n"," resposta_prevista \n","0 Answer according to: Article 5: This law shall... \n","1 Sure, the publicly issued interpertation guida... \n","2 Answer according to… The act specifies that th... \n","3 Yes, field assignments are subject to applicat... \n","4 Yes, a university can issue a degree with spec... \n",".. ... \n","96 Resolution Nº 027/2010-CEG came into effect on... \n","97 A student receives a mention of 'dispensed' in... \n","98 No, any institution recognized by the Brazilia... \n","99 No, credit will only be awarded if the student... \n","100 In the Ministry of Education's office in the c... \n","\n","[101 rows x 3 columns]\n"]}]},{"cell_type":"code","source":["results_df['resposta_prevista']"],"metadata":{"id":"UL9zyqoiSk8h","colab":{"base_uri":"https://localhost:8080/"},"outputId":"61c00ee7-d541-4f2d-80b7-35805614ce76"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0 Answer according to: Article 5: This law shall...\n","1 Sure, the publicly issued interpertation guida...\n","2 Answer according to… The act specifies that th...\n","3 Yes, field assignments are subject to applicat...\n","4 Yes, a university can issue a degree with spec...\n"," ... \n","96 Resolution Nº 027/2010-CEG came into effect on...\n","97 A student receives a mention of 'dispensed' in...\n","98 No, any institution recognized by the Brazilia...\n","99 No, credit will only be awarded if the student...\n","100 In the Ministry of Education's office in the c...\n","Name: resposta_prevista, Length: 101, dtype: object"]},"metadata":{},"execution_count":29}]},{"cell_type":"markdown","source":["### **Similaridade do Cosseno**\n","\n","Para avaliar o quão próximas estão as respostas do modelo Tiny Llama com LORA em relação às respostas reais da base de dados sintética, vamos usar a similaridade do cosseno. Para fazer isso, vamos usar o **modelo de linguagem SentenceTransformer com o modelo all-mpnet-base-v2**, que permite **criar embeddings** das palavras para calcular a similaridade do cosseno. Essa métrica é fundamental para **medir a similaridade entre duas sequências de texto**, transformando-as em vetores e calculando o cosseno do ângulo entre eles. A similaridade do cosseno varia de -1 a 1, onde 1 indica uma correspondência perfeita, 0 indica que não há relação e -1 indica que são completamente opostas.\n","\n","No contexto do nosso modelo Tiny Llama com LORA, a similaridade do cosseno vai nos ajudar a quantificar a precisão e relevância das respostas geradas, oferecendo uma medida de desempenho que vai nos guiar a possíveis melhorias que podem ser feitas no modelo.\n","\n","Para realizar essa avaliação, seguimos os seguintes passos:\n","\n","- **Carregar o modelo de embeddings**: Utilizamos o SentenceTransformer para transformar frases em vetores numéricos.\n","- **Processar os dados**: Carregamos o arquivo csv com as respostas reais e previstas que criamos anteriormente.\n","- **Calcular a similaridade**: Para cada par de respostas (real e prevista), calculamos a similaridade do cosseno e adicionamos essa informação ao dataframe.\n","- **Analisar os resultados**: Calculamos a média das similaridades para obter uma visão geral do desempenho do modelo."],"metadata":{"id":"9ijJaTl4vAFL"}},{"cell_type":"markdown","source":["Vamos ver abaixo alguns exemplos do cálculo da similaridade do cosseno para algumas respostas reais e previstas do dataframe que acabamos de criar."],"metadata":{"id":"5zQybIfp2EyU"}},{"cell_type":"code","source":["# carregando o modelo\n","model = SentenceTransformer('all-mpnet-base-v2')"],"metadata":{"id":"q41gNixh4Iot"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# exemplo 1\n","real = \"Yes, Deaf individuals receive priority admission into courses related to Libras instruction and formation\"\n","prevista = \"Yes, Deaf individuals have priority access to certain courses offered by the university\"\n","\n","# criação dos embeddings\n","embedding1 = model.encode(real, convert_to_tensor=True)\n","embedding2 = model.encode(prevista, convert_to_tensor=True)\n","\n","# cálculo da similaridade do cosseno\n","cosine_similarity = util.pytorch_cos_sim(embedding1, embedding2)\n","\n","print(\"Similaridade de Cosseno:\", cosine_similarity.item())"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"6vLAmO4b0dkR","executionInfo":{"status":"ok","timestamp":1722949998576,"user_tz":180,"elapsed":621,"user":{"displayName":"Bianka Vasconcelos","userId":"00512411367206176161"}},"outputId":"d9f4e572-5d53-4a2f-d61c-d8bebc14b5c9"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Similaridade de Cosseno: 0.8298245668411255\n"]}]},{"cell_type":"code","source":["# exemplo 2\n","real = \"Yes, being an author or co-author of a chapter in a book is recognized as a complementary activity according to Art. 4 Item IV.\"\n","prevista = \"Yes, authors or co-authors of published book chapters are eligible for recognition as complementary activities.\"\n","\n","# criação dos embeddings\n","embedding1 = model.encode(real, convert_to_tensor=True)\n","embedding2 = model.encode(prevista, convert_to_tensor=True)\n","\n","# cálculo da similaridade do cosseno\n","cosine_similarity = util.pytorch_cos_sim(embedding1, embedding2)\n","\n","print(\"Similaridade de Cosseno:\", cosine_similarity.item())"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ozXzJTHW1fEW","executionInfo":{"status":"ok","timestamp":1722950249109,"user_tz":180,"elapsed":523,"user":{"displayName":"Bianka Vasconcelos","userId":"00512411367206176161"}},"outputId":"a0e026f6-527f-4c96-d543-70a775005311"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Similaridade de Cosseno: 0.832678496837616\n"]}]},{"cell_type":"code","source":["# exemplo 3\n","real = \"To ensure consistency in integrating undergraduate courses taken by students across different departments.\"\n","prevista = \"Because it is necessary to ensure the proper functioning of the system of higher education in the country.\"\n","\n","# criação dos embeddings\n","embedding1 = model.encode(real, convert_to_tensor=True)\n","embedding2 = model.encode(prevista, convert_to_tensor=True)\n","\n","# cálculo da similaridade do cosseno\n","cosine_similarity = util.pytorch_cos_sim(embedding1, embedding2)\n","\n","print(\"Similaridade de Cosseno:\", cosine_similarity.item())"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"c199_Tg22MtU","executionInfo":{"status":"ok","timestamp":1722950556771,"user_tz":180,"elapsed":514,"user":{"displayName":"Bianka Vasconcelos","userId":"00512411367206176161"}},"outputId":"83ad0458-8b89-4e46-aa96-e42c6f51c814"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Similaridade de Cosseno: 0.5239999294281006\n"]}]},{"cell_type":"markdown","source":["Agora, vamos aplicar a similaridade do cosseno para todas as respostas reais e previstas do dataframe."],"metadata":{"id":"OIMrRUq92y8R"}},{"cell_type":"code","source":["# carregando o csv\n","df = pd.read_csv('results.csv', sep='\\t')\n","\n","# alguns valores o modelo não conseguiu prever, então filtramos as colunas vazias\n","df['resposta_real'] = df['resposta_real'].fillna('')\n","df['resposta_prevista'] = df['resposta_prevista'].fillna('')\n","\n","# Função para calcular a similaridade do cosseno\n","def calcular_similaridade(row):\n"," frase1 = str(row['resposta_real'])\n"," frase2 = str(row['resposta_prevista'])\n","\n"," # Obter embeddings das frases\n"," embedding1 = model.encode(frase1, convert_to_tensor=True)\n"," embedding2 = model.encode(frase2, convert_to_tensor=True)\n","\n"," # Calcular similaridade do cosseno\n"," similaridade = util.pytorch_cos_sim(embedding1, embedding2).item()\n"," return similaridade"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ZDjvP8PDsv32","executionInfo":{"status":"ok","timestamp":1722948256291,"user_tz":180,"elapsed":5291,"user":{"displayName":"Bianka Vasconcelos","userId":"00512411367206176161"}},"outputId":"f4a8edae-b40d-4f92-aa6b-887d3d7953ff"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stderr","text":["/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n","The secret `HF_TOKEN` does not exist in your Colab secrets.\n","To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n","You will be able to reuse this secret in all of your notebooks.\n","Please note that authentication is recommended but still optional to access public models or datasets.\n"," warnings.warn(\n"]}]},{"cell_type":"code","source":["# aplicar em cada linha do df\n","df['similaridade_cosseno'] = df.apply(calcular_similaridade, axis=1)\n","\n","# média da similaridade do cosseno\n","media_similaridade = df['similaridade_cosseno'].mean()\n","\n","# salva no df\n","df.to_csv('resultados_com_similaridade.csv', sep='\\t', index=False)\n","\n","# mostra o df\n","print(df[['resposta_real', 'resposta_prevista', 'similaridade_cosseno']])"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"IYXpr6APpK7X","executionInfo":{"status":"ok","timestamp":1722948309931,"user_tz":180,"elapsed":8734,"user":{"displayName":"Bianka Vasconcelos","userId":"00512411367206176161"}},"outputId":"b49e0c8d-39cb-45ae-c98f-be2793dd6486"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":[" resposta_real \\\n","0 This law becomes effective thirty days from it... \n","1 The listed government officials authored rule ... \n","2 Granting falls upon coordinators' duties with ... \n","3 Field works aren't explicitly mentioned being ... \n","4 No, they cannot. According to the response fro... \n",".. ... \n","96 Resolution Nº 027/2010-CEG came into effect on... \n","97 A student who registered but wasn't selected b... \n","98 Foreign higher education institutions may also... \n","99 Meeting the minimal percentage thresholds alon... \n","100 Requests for accreditation from institutions w... \n","\n"," resposta_prevista similaridade_cosseno \n","0 Answer according to: Article 5: This law shall... 0.754316 \n","1 Sure, the publicly issued interpertation guida... 0.540074 \n","2 Answer according to… The act specifies that th... 0.422281 \n","3 Yes, field assignments are subject to applicat... 0.456716 \n","4 Yes, a university can issue a degree with spec... 0.553797 \n",".. ... ... \n","96 Resolution Nº 027/2010-CEG came into effect on... 0.943615 \n","97 A student receives a mention of 'dispensed' in... 0.751552 \n","98 No, any institution recognized by the Brazilia... 0.455082 \n","99 No, credit will only be awarded if the student... 0.250302 \n","100 In the Ministry of Education's office in the c... 0.451453 \n","\n","[101 rows x 3 columns]\n"]}]},{"cell_type":"code","source":["print(f'Média da similaridade do cosseno: {media_similaridade}')"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"QaHu_H-3tZ3b","executionInfo":{"status":"ok","timestamp":1722948321410,"user_tz":180,"elapsed":501,"user":{"displayName":"Bianka Vasconcelos","userId":"00512411367206176161"}},"outputId":"ae59dafc-43e4-4c56-f7fa-a5a31f35378a"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Média da similaridade do cosseno: 0.4656926949720572\n"]}]},{"cell_type":"markdown","source":["Como podemos observar, o valor médio da similaridade do cosseno foi, de 0.46, o que sugere que há um grau de similaridade entre as frases reais e previstas, mas elas **não são totalmente semelhantes**. Isso significa que, apesar de o modelo trazer resultados diferentes, ele consegue identificar a essência contextual da pergunta.\n","\n","Possíveis melhorias para aumentar a similaridade do cosseno seria realizar o Fine Tuning com LORA em um modelo melhor, o que exigiria mais desempenho do ambiente de execução. Outra solução é aumentar a quantidade de exemplos da base de dados sintética de instruções, visando dar mais contexto para ser mais fácil para o modelo conseguir generalizar."],"metadata":{"id":"JFnCNV-2tVAk"}}]}