rch-rise-250821-p01 - Deep Analysis & Implementation Plan: Contextual Transposition Intelligence System
Approaches to enhancing language model capabilities for complex tasks involve various prompting strategies and architectural modifications. Chain-of-Thought (CoT) prompting is a widely explored technique that enables models to demonstrate sequential reasoning by breaking down complex queries into intermediary steps, which generally increases solution accuracy [24, 50, 66]. This foundational concept has been extended to include Chain-of-Thought Factored Decomposition, which further refines the model's analytical process by guiding it to sequentially address subcomponents of a task [114, 115, 116]. Other CoT variants support specialized reasoning, such as reverse chain-of-thought for geometric problems [1], structured chain-of-thought for code generation [9], chain-of-symbol for planning [5], chain-of-knowledge for grounding models with dynamic knowledge [10], and chain-of-table for table understanding [23].
Beyond linear reasoning, more complex structures have been introduced, including Tree-of-Thoughts (ToT) [13, 117] and Graph-of-Thoughts (GoT) [94], which allow models to explore diverse and creative pathways by forming decision trees or graph structures. Layer-of-Thoughts prompting also leverages retrieval with constraint hierarchies [3]. To facilitate complex reasoning, methods like "Show Your Work" utilize scratchpads for intermediate computations [16], while successive prompting focuses on decomposing complex questions [102]. Techniques like least-to-most prompting enable complex reasoning by breaking problems into a series of simpler steps [134], and plan-and-solve prompting enhances zero-shot CoT reasoning [123]. Program-aided language models (PAL) integrate explicit programming logic to disentangle computation from reasoning, particularly for numerical tasks [96, 104].
For robustness and reliability, mechanisms for self-correction and refinement are crucial. Self-refine approaches employ iterative refinement with self-feedback [14], and a "Critic" mechanism allows models to self-correct with tool-interactive critiquing [107]. Self-consistency methods further improve CoT reasoning by sampling multiple reasoning paths and selecting the most consistent answer [22, 113]. Addressing potential issues, Chain-of-Verification seeks to reduce hallucination [72], and comprehensive surveys exist on hallucination and its mitigation techniques in large language models [20, 21].
The application of these reasoning strategies extends to code generation and automated software engineering. Models are trained on code and evaluated for their coding abilities [95], with structured chain-of-thought being applied for code generation [9]. Code prompting can elicit conditional reasoning abilities in text+code models [18], and tools like "Chain of Code" facilitate reasoning with language model-augmented code emulators [8]. There is a growing understanding that prompts themselves can be viewed as programs, influencing how developers interact with LLMs in software development [86]. Automated tools and agents, such as SWE-agent [56] and Devin [52], are emerging to automate software engineering tasks. However, challenges persist, including the non-deterministic nature of large language models in code generation [33, 90] and issues related to prompt evolution, such as insufficient documentation and logical inconsistencies [75]. The usability of AI programming assistants is also an active area of survey [87].
The development of sophisticated AI systems also necessitates formal grounding and rigorous validation. Logical reasoning is being injected into contexts through "Logic-of-Thought" [12] and explored using first-order logic frameworks [109]. Surveys on logical reasoning over natural language highlight its importance as knowledge representation [126]. The concept of "verifiers" can be trained to solve problems and assist in interpretations [99, 100]. Evaluating the instruction-following ability of models is an ongoing research area [37, 64]. Furthermore, aligning LLM-assisted evaluation with human preferences is a critical aspect of validation [44, 146]. Tools for prompt optimization, such as GRAD-SUM [142], Promptbreeder [77], and DSPy [46], are being developed to systemize prompt refinement, although LLM sensitivity to prompt revisions remains an area of ongoing study [144].
The broader field of prompt engineering is rapidly evolving, necessitating comprehensive overviews and structured frameworks. Systematic surveys provide taxonomies of prompting methods [11, 65, 69, 98, 112, 116, 130], identifying key techniques and design dimensions [67]. Catalogs of prompt patterns are also being developed to enhance the discipline of prompt engineering [41]. Tools such as ChainForge offer visual toolkits for prompt engineering and hypothesis testing [141]. These frameworks aim to bridge the gap between academic research and practical application by consolidating diverse techniques into accessible tools like the Prompt Canvas [2]. The shift towards multi-modal models [26] and agent-based systems [122, 125, 135] represents future directions, involving complex interactions, implicit user intention understanding [35], and the potential for emotional stimuli to enhance LLMs [7].
[1] Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2024). R-cot: Reverse chain-of-thought problem generation for geometric reasoning in large multimodal models. [2] Braun, M., Greve, M., Kegel, F., Kolbe, L., & Beyer, P. E. (2024). Can (a)i have a word with you? a taxonomy on the design dimensions of AI prompts. In Proceedings of the 57th Hawaii International Conference on System Sciences | 2024. [3] Fungwacharakorn, W., Thanh, N. H., Zin, M. M., & Satoh, K. (2024). Layer-of-thoughts prompting (lot): Leveraging llm-based retrieval with constraint hierarchies. [4] Hong, B., Wu, J., Liu, J., Ding, L., Sha, J., Zhang, K., Wang, S., & Huang, Z. (2024). End-to-end graph flattening method for large language models. [5] Hu, H., Lu, H., Zhang, H., Song, Y. Z., Lam, W., & Zhang, Y. (2023). Chain-of-symbol prompting elicits planning in large langauge models. [6] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Kuttler, H., Lewis, M., Yih, W. T., Rocktaschel, T., et al. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33, 9459–9474. [7] Li, C., Wang, J., Zhang, Y., Zhu, K., Hou, W., Lian, J., Luo, F., Yang, Q., & Xie, X. (2023). Large language models understand and can be enhanced by emotional stimuli. arXiv preprint arXiv:2307.11760. [8] Li, C., Liang, J., Zeng, A., Chen, X., Hausman, K., Sadigh, D., Levine, S., Fei-Fei, L., Xia, F., & Ichter, B. (2023). Chain of code: Reasoning with a language model-augmented code emulator. arXiv preprint arXiv:2312.04474. [9] Li, J., Li, G., Li, Y., & Jin, Z. (2023). Structured chain-of-thought prompting for code generation. arXiv preprint arXiv:2305.06599. [10] Li, X., Zhao, R., Chia, Y. K., Ding, B., Joty, S., Poria, S., & Bing, L. (2023). Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. [11] Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9), 1–35. [12] Liu, T., Xu, W., Huang, W., Wang, X., Wang, J., Yang, H., & Li, J. (2024). Logic-of-thought: Injecting logic into contexts for full reasoning in large language models. [13] Long, J. (2023). Large language model guided tree-of-thought. arXiv preprint arXiv:2305.08291. [14] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Welleck, S., Majumder, B. P., Gupta, S., Yazdanbakhsh, A., & Clark, P. (2023). Self-refine: Iterative refinement with self-feedback. [15] Mekala, R. R., Razeghi, Y., & Singh, S. (2024). Echoprompt: Instructing the model to rephrase queries for improved in-context learning. [16] Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021). Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114. [17] Paranjape, B., Lundberg, S., Singh, S., Hajishirzi, H., Zettlemoyer, L., & Ribeiro, M. T. (2023). Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014. [18] Puerto, H., Tutek, M., Aditya, S., Zhu, X., & Gurevych, I. (2024). Code prompting elicits conditional reasoning abilities in text+code llms. [19] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9. [20] Sahoo, P., Meharia, P., Ghosh, A., Saha, S., Jain, V., & Chadha, A. (2024). A comprehensive survey of hallucination in large language, image, video and audio foundation models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11709–11724. [21] Tonmoy, S. M., Zaman, S. M., Jain, V., Rani, A., Rawte, V., Chadha, A., & Das, A. (2024). A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313. [22] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. [23] Wang, Z., Zhang, H., Li, C. L., Eisenschlos, J. M., Perot, V., Wang, Z., Miculicich, L., Fujii, Y., Shang, J., Lee, C. Y., & Pfister, T. (2024). Chain-of-table: Evolving tables in the reasoning chain for table understanding. [24] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824–24837. [25] Weston, J., & Sukhbaatar, S. (2023). System 2 attention (is something you might need too). arXiv preprint arXiv:2311.11829. [26] Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., & Duan, N. (2023). Visual chatgpt: Talking, drawing and editing with visual foundation models. [27] Xu, S., Xie, W., Zhao, L., & He, P. (2025). Chain of draft: Thinking faster by writing less. [28] Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2023). In-IDE Generation-based Information Support with a Large Language Model. arXiv [cs.SE] arXiv:2307.08177. [29] Neto, V. L., Coelho, R., Leite, L., Guerrero, D. S., & Mendonça, A. P. (2013). POPT: A Problem-Oriented Programming and Testing approach for novice students. In 2013 35th International Conference on Software Engineering (ICSE), IEEE, 1099–1108. [30] Nguyen, S., Babe, H. M., Zi, Y., Guha, A., Anderson, C. J., & Feldman, M. Q. (2024). How Beginning Programmers and Code LLMs (Mis)read Each Other. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24, Article 651), Association for Computing Machinery, New York, NY, USA, 1–26. [31] OpenAI. (2023). GPT-4 Technical Report. [32] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, 27730–27744. [33] Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2023). LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation. arXiv preprint arXiv:2308.02828. [34] Petridis, S., Wedin, B. D., Wexler, J., Pushkarna, M., Donsbach, A., Goyal, N., Cai, C. J., & Terry, M. (2024). Constitution-maker: Interactively critiquing large language models by converting feedback into principles. In Proceedings of the 29th International Conference on Intelligent User Interfaces, 853–868. [35] Qian, C., He, B., Zhuang, Z., Deng, J., Qin, Y., Cong, X., Zhang, Z., Zhou, J., Lin, Y., Liu, Z., & Sun, M. (2024). Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents. arXiv [cs.CL] arXiv:2402.09205. [36] Qian, C., Zhao, X., & Wu, S. T. (2023). “merge conflicts!” exploring the impacts of external distractors to parametric knowledge graphs. arXiv [cs.CL] arXiv:2309.08594. [37] Qin, Y., Song, K., Hu, Y., Yao, W., Cho, S., Wang, X., Wu, X., Liu, F., Liu, P., & Yu, D. (2024). InFoBench: Evaluating Instruction Following Ability in Large Language Models. arXiv [cs.CL] arXiv:2401.03601. [38] Radermacher, A., Walia, G., & Knudson, D. (2014). Investigating the skill gap between graduating students and industry expectations. In Companion Proceedings of the 36th International Conference on Software Engineering (ICSE Companion 2014), Association for Computing Machinery, New York, NY, USA, 291–300. [39] Raza, M., Gulwani, S., & Milic-Frayling, N. (2015). Compositional program synthesis from natural language and examples. In Proceedings of the 24th International Conference on Artificial Intelligence (IJCAI’15), AAAI Press, 792–800. [40] Rogaten, J., Rienties, B., Sharpe, R., Cross, S., Whitelock, D., Lygo-Baker, S., & Littlejohn, A. (2019). Reviewing affective, behavioural and cognitive learning gains in higher education. Assessment & Evaluation in Higher Education, 44(3), 321–337. [41] Schmidt, D. C., Spencer-Smith, J., Fu, Q., & White, J. (2024). Towards a catalog of prompt patterns to enhance the discipline of prompt engineering. ACM SIGAda Ada Lett., 43(2), 43–51. [42] Shani, I. (2023). Survey reveals AI’s impact on the developer experience. https://github.blog/news-insights/research/survey-reveals-ais-impact-on-the-developer-experience/. [43] Shankar, S., Li, H., Asawa, P., Hulsebos, M., Lin, Y., Zamfirescu-Pereira, J. D., Chase, H., Fu-Hinthorn, W., Parameswaran, A. G., & Wu, E. (2024). SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines. arXiv [cs.DB] arXiv:2401.03038. [44] Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. arXiv [cs.HC] arXiv:2404.12272. [45] Shin, T., Razeghi, Y., Logan, R. L., IV, Wallace, E., & Singh, S. (2020). Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980. [46] Singhvi, A., Shetty, M., Tan, S., Potts, C., Sen, K., Zaharia, M., & Khattab, O. (2023). DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines. arXiv [cs.CL] arXiv:2312.13382. [47] Su, D., Zhao, Y., Hou, X., Wang, S., & Wang, H. (2024). Gpt store mining and analysis. arXiv preprint arXiv:2405.10210. [48] Walia, G. S., & Carver, J. C. (2009). A systematic literature review to identify and classify software requirement errors. Information and Software Technology, 51(7), 1087–1109. [49] Wang, Q., Walsh, S., Si, M., Kephart, J., Weisz, J. D., & Goel, A. K. (2024). Theory of Mind in Human-AI Interaction. In Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (CHI EA ’24), Association for Computing Machinery, New York, NY, USA, Article 493, 6 pages. [50] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824–24837. [51] Wiggins, G. P., & McTighe, J. (2005). Understanding by Design. ASCD. [52] Wodecki, B. (2024). Devin: AI Software Engineer that Codes Entire Projects from Single Prompt. https://aibusiness.com/automation/ai-software-engineer-devin-codes-entire-projects-from-single-prompt. [53] Wu, Y., Fan, Y., Min, S. Y., Prabhumoye, S., McAleer, S., Bisk, Y., Salakhutdinov, R., Li, Y., & Mitchell, T. (2024). AgentKit: Flow Engineering with Graphs, not Coding. arXiv [cs.AI] arXiv:2404.11483. [54] Yang, C., Rustogi, R., Brower-Sinning, R., Lewis, G., Kaestner, C., & Wu, T. (2023). Beyond Testers’ Biases: Guiding Model Testing with Knowledge Bases using LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.), Association for Computational Linguistics, Singapore, 13504–13519. [55] Yang, C., Zhuang, Y., Zhang, J., Meng, Y., Hong, Y., Lewis, G. A., Wu, T., & Kaestner, C. (2024). What Is Wrong with My Model? Identifying Systematic Problems with Semantic Data Slicing. In the 39th IEEE/ACM International Conference on Automated Software Engineering. [56] Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv:2405.15793. [57] Ym, P., Ganesan, V., Arumugam, D. K., Gupta, M., Shadagopan, N., Dixit, T., Segal, S., Kumar, P., Jain, M., & Rajamani, S. (2023). PwR: Exploring the role of representations in conversational programming. arXiv [cs.HC] arXiv:2309.09495. [58] Yu, Y., Zhuang, Y., Zhang, J., Meng, Y., Ratner, A. J., Krishna, R., Shen, J., & Zhang, C. (2024). Large language model as attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems, 36. [59] Yuan, W., Liu, P., & Gallé, M. (2024). LLMCRIT: Teaching Large Language Models to Use Criteria. arXiv [cs.CL] arXiv:2403.01069. [60] Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z., Guestrin, C., & Zou, J. (2024). TextGrad: Automatic “Differentiation” via Text. arXiv [cs.CL] arXiv:2406.07496. [61] Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023). Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23, Article 437), Association for Computing Machinery, New York, NY, USA, 1–21. [62] Zhang, Z., Zhang, L., Yuan, X., Zhang, A., Xu, M., & Qian, F. (2024). A first look at gpt apps: Landscape and vulnerability. arXiv preprint arXiv:2402.15105. [63] Zhao, L., Alhoshan, W., Ferrari, A., Letsholo, K. J., & others. (2021). Natural language processing for requirements engineering: A systematic mapping study. ACM Computing. [64] Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., & Hou, L. (2023). Instruction-following evaluation for Large Language Models. arXiv [cs.CL] arXiv:2311.07911. [65] Bhandari, P. (2024). A survey on prompting techniques in LLMs. URL https://arxiv.org/abs/2312.03740. [66] Bozkurt, A. (2024). Tell me your prompts and i will make them true: The alchemy of prompt engineering and generative AI. Open Praxis, 16(2), 111–118. [67] Braun, M., Greve, M., Kegel, F., Kolbe, L., & Beyer, P. E. (2024). Can (a)i have a word with you? a taxonomy on the design dimensions of AI prompts. In Proceedings of the 57th Hawaii International Conference on System Sciences | 2024. [68] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language models are few-shot learners. [69] Chen, B., Zhang, Z., Langrené, N., & Zhu, S. (2024). Unleashing the potential of prompt engineering in large language models: A comprehensive review. CoRR, abs/2310.14735. [70] Chong, L., Lo, I. P., Rayan, J., Dow, S., Ahmed, F., & Lykourentzou, I. (2024). Prompting for products: Investigating design space exploration strategies for text-to-image generative models. arXiv https://arxiv.org/abs/2408.03946v1. [71] Cooper, H. M. (1988). Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society, 1(1). [72] Diao, S., Wang, P., Lin, Y., & Zhang, T. (2023). Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246. [73] Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., Stewart, L. A., Thomas, J., Tricco, J. C., Welch, B. H., Whiting, P., & Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372, n71. [74] Aguiar, L., Paixao, M., Carmo, R., Soares, E., Leal, A., Freitas, M., & Gama, E. (2024). Multi-language software development in the llm era: Insights from practitioners’ conversations with chatgpt. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ser. ESEM ’24, New York, NY, USA: Association for Computing Machinery, 489–495. [75] Cai, Y., Liang, P., Wang, Y., Li, Z., & Shahin, M. (2024). Demystifying issues, causes and solutions in llm open-source projects. arXiv preprint arXiv:2409.16559. [76] Desmond, M., & Brachman, M. (2024). Exploring prompt engineering practices in the enterprise. arXiv preprint arXiv:2403.08950. [77] Fernando, C., Banarse, D., Michalewski, H., Osindero, S., & Rocktaschel, T. (2023). Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797. [78] Gao, S., Wen, X. C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023). What makes good in-context demonstrations for code intelligence tasks with llms? In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE, 761–773. [79] Hassan, A. E., Lin, D., Rajbahadur, G. K., Gallaba, K., Cogo, F. R., Chen, B., Zhang, H., Thangarajah, K., Oliva, G., Lin, J. J., Abdullah, W. M., & Jiang, Z. M. J. (2024). Rethinking software engineering in the era of foundation models: A curated catalogue of challenges in the development of trustworthy fmware. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, ser. FSE 2024, New York, NY, USA: Association for Computing Machinery, 294–305. [80] Hu, T., & Collier, N. (2024). Quantifying the persona effect in llm simulations. arXiv preprint arXiv:2402.10811. [81] Imani, A., Ahmed, I., & Moshirpour, M. (2024). Context conquers parameters: Outperforming proprietary llm in commit message generation. arXiv preprint arXiv:2408.02502. [82] Jiang, F., Xu, Z., Niu, L., Wang, B., Jia, J., Li, B., & Poovendran, R. (2023). Identifying and mitigating vulnerabilities in llm-integrated applications. arXiv preprint arXiv:2311.16153. [83] Li, J., Faragó, D., Petrov, C., & Ahmed, I. (2024). Only diff is not enough: Generating commit messages leveraging reasoning and action of large language model. Proceedings of the ACM on Software Engineering, 1(FSE), 745–766. [84] Li, J., Raheja, V., & Kumar, D. (2023). Contradoc: Understanding self-contradictions in documents with large language models. arXiv preprint arXiv:2311.09182. [85] Li, Y. B., & Wu, K. (2023). Spell: Semantic prompt evolution based on a llm. arXiv preprint arXiv:2310.01260. [86] Liang, J. T., Lin, M., Rao, N., & Myers, B. A. (2024). Prompts are programs too! understanding how developers build software containing prompts. arXiv preprint arXiv:2409.12447. [87] Liang, J. T., Yang, C., & Myers, B. A. (2024). A large-scale survey on the usability of ai programming assistants: Successes and challenges. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24, New York, NY, USA: Association for Computing Machinery. [88] Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837. [89] Nahar, N., Kästner, C., Butler, J., Parnin, C., Zimmermann, T., & Bird, C. (2024). Beyond the comfort zone: Emerging solutions to overcome challenges in integrating llms into software products. arXiv preprint arXiv:2410.12071. [90] Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2024). An empirical study of the non-determinism of chatgpt in code generation. ACM Transactions on Software Engineering and Methodology. [91] Abercrombie, G., Curry, A., Dinkar, T., Rieser, V., & Talat, Z. (2023). Mirages. on anthropomorphism in dialogue systems. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 4776–4790, Singapore. Association for Computational Linguistics. [92] Amatriain, X. (2024). Prompt design and engineering: Introduction and advanced methods. arXiv preprint arXiv:2401.14423. [93] Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021). Program synthesis with large language models. arXiv preprint arXiv:2108.07732. [94] Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Gianinazzi, L., Gajda, J., Lehmann, T., Podstawski, M., Niewiadomski, H., Nyczyk, P., et al. (2023). Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687. [95] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. [96] Chen, W., Ma, X., Wang, X., & Cohen, W. W. (2022). Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. [97] Chen, X., Lin, M., Schärli, N., & Zhou, D. (2023). Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128. [98] Chu, Z., Chen, J., Chen, Q., Yu, W., He, T., Wang, H., Peng, W., Liu, M., Qin, B., & Liu, T. (2023). A survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309.15402. [99] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. [100] Creswell, A., Shanahan, M., & Higgins, I. (2022). Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Representations. [101] Deng, Y., Lei, W., Liao, L., & Chua, T. S. (2023). Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration. arXiv preprint arXiv:2305.13626. [102] Dua, D., Gupta, S., Singh, S., & Gardner, M. (2022). Successive prompting for decomposing complex questions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 1251–1265. [103] Fischer, K. A. (2023). Reflective linguistic programming (rlp): A stepping stone in socially-aware agi (socialagi). arXiv preprint arXiv:2305.12647. [104] Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G. (2023). PAL: Program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, 10764–10799. PMLR. [105] Geng, S., Liu, S., Fu, Z., Ge, Y., & Zhang, Y. (2022). Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference on Recommender Systems, 299–315. [106] Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021). Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9, 346–361. [107] Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023). Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738. [108] Gramopadhye, M., & Szafir, D. (2023). Generating executable action plans with environmentally-aware language models. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 3568–3575. IEEE. [109] Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Zhou, W., Coady, J., Peng, D., Qiao, Y., Benson, L., et al. (2022). Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.00840. [110] Hao, S., Gu, Y., Ma, H., Hong, J. J., Wang, Z., Wang, D. Z., & Hu, Z. (2023). Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992. [111] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Gu, S. (2022). Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916. [112] Lyu, Q., Huang, Y., Zhang, W., Chen, B., Li, X., Wu, X., & Zhou, B. (2023). Chain-of-Thought prompting in large language models: A review. arXiv preprint arXiv:2307.03157. [113] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations. [114] Wang, Y., Jiang, Z., Chen, Z., Yang, F., Zhou, Y., Cho, E., Fan, X., Huang, X., Lu, Y., & Yang, Y. (2022). Zero-shot chain of thought: Enhancing LLM reasoning without examples. arXiv preprint arXiv:2211.08411. [115] Fu, J., Sun, M., Liu, P., & Peng, Z. (2022). Zero-shot chain of thought reasoning: Rethinking the power of LLMs. arXiv preprint arXiv:2205.11916. [116] Xue, H., Cui, S., Chen, S., Zhang, S., Liang, H., Lu, W., & Xu, Z. (2023). Chain-of-thought prompting for large language models: A comprehensive review. arXiv preprint arXiv:2309.00315. [117] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. [118] Tufano, R., Dabić, O., Mastropaolo, A., Ciniselli, M., & Bavota, G. (2024). Code review automation: Strengths and weaknesses of the state of the art. IEEE Transactions on Software Engineering. [119] Wang, H., Liu, Z., Wang, S., Cui, G., Ding, N., Liu, Z., & Yu, G. (2023). Intervenor: Prompt the coding ability of large language models with the interactive chain of repairing. arXiv preprint arXiv:2311.09868. [120] Wang, H., Hu, M., Deng, Y., Wang, R., Mi, F., Wang, W., Wang, Y., Kwan, W. C., King, I., & Wong, K. F. (2023). Large language models as source planner for personalized knowledge-grounded dialogue. arXiv preprint arXiv:2310.08840. [121] Wang, H., Wang, R., Mi, F., Deng, Y., Wang, Z., Liang, B., Xu, R., & Wong, K. F. (2023). Cue-cot: Chain-of-thought prompting for responding to in-depth dialogue questions with llms. arXiv preprint arXiv:2305.11792. [122] Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024). A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), 186345. [123] Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K., & Lim, E. P. (2023). Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04091. [124] Wang, Y., Jiang, Z., Chen, Z., Yang, F., Zhou, Y., Cho, E., Fan, X., Huang, X., Lu, Y., & Yang, Y. (2023). Recmind: Large language model powered agent for recommendation. arXiv preprint arXiv:2308.14296. [125] Wang, Z., Cai, S., Liu, A., Ma, X., & Liang, Y. (2023). Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560. [126] Yang, Z., Du, X., Mao, R., Ni, J., & Cambria, E. (2023). Logical reasoning over natural language as knowledge representation: A survey. arXiv preprint arXiv:2303.12023. [127] Yao, J., Xu, W., Lian, J., Wang, X., Yi, X., & Xie, X. (2023). Knowledge plugins: Enhancing large language models for domain-specific recommendations. arXiv preprint arXiv:2311.10779. [128] Yoran, O., Wolfson, T., Bogin, B., Katz, U., Deutch, D., & Berant, J. (2023). Answering questions by meta-reasoning over multiple chains of thought. arXiv preprint arXiv:2304.13007. [129] Yu, X., Chen, M., & Yu, Z. (2023). Prompt-based monte-carlo tree search for goal-oriented dialogue policy planning. arXiv preprint arXiv:2305.13660. [130] Yu, Z., He, L., Wu, Z., Dai, X., & Chen, J. (2023). Towards better chain-of-thought prompting strategies: A survey. arXiv preprint arXiv:2310.04959. [131] Zhang, M., Wang, Z., Yang, Z., Feng, W., & Lan, A. (2023). Interpretable math word problem solution generation via step-by-step planning. arXiv preprint arXiv:2306.00784. [132] Zhao, R., Li, X., Joty, S., Qin, C., & Bing, L. (2023). Verify-and-edit: A knowledge-enhanced chain-of-thought framework. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5823–5840, Toronto, Canada. Association for Computational Linguistics. [133] Zhao, Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021). Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, 12697–12706. PMLR. [134] Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. V., et al. (2022). Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations. [135] Zhu, X., Chen, Y., Tian, H., Tao, C., Su, W., Yang, C., Huang, G., Li, B., Lu, L., Wang, X., et al. (2023). Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305.17143. [136] Abassi, L., & Boukhris, I. (2017). A gold standards-based crowd label aggregation within the belief function theory. In Advances in Artificial Intelligence: From Theory to Practice: 30th International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2017, Arras, France, June 27-30, 2017, Proceedings, Part II 30. Springer, 97–106. [137] Abassi, L., & Boukhris, I. (2019). A worker clustering-based approach of label aggregation under the belief function theory. Applied Intelligence, 49, 53–62. [138] Abdalla, M., & Fine, B. (2023). Hurdles to artificial intelligence deployment: noise in schemas and “gold” labels. Radiology: Artificial Intelligence, 5(2), e220056. [139] Austin, I., Alonso, O., Kandylas, V., Patel, R., Shelford, S., & Slivkins, A. (2016). How many workers to ask? Adaptive exploration for collecting high quality labels. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 473–482. [140] Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019). Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems. 1–13. [141] Arawjo, I., Swoopes, C., Vaithilingam, P., Wattenberg, M., & Glassman, E. L. (2024). ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–18. [142] Austin, D., & Chartock, E. (2024). GRAD-SUM: Leveraging Gradient Summarization for Optimal Prompt Engineering. arXiv preprint arXiv:2407.12865. [143] Bernard, J., Zeppelzauer, M., Sedlmair, M., & Aigner, W. (2018). VIAL: a unified process for visual interactive labeling. The Visual Computer, 34, 1189–1207. [144] Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-computer Interaction, 5(CSCW1), 1–21. [145] Caiafa, C. F., Solé-Casals, J., Marti-Puig, P., Zhe, S., & Tanaka, T. (2020). Decomposition methods for machine learning with small, incomplete or noisy datasets. Applied Sciences, 10(23), 8481. [146] Chiang, C. H., & Lee, H. Y. (2023). Can Large Language Models Be an Alternative to Human Evaluations?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.), Association for Computational Linguistics, Toronto, Canada, 15607–15631.