Prompt Engineering and LLM Resilience for Software and Data Generation

Abstract

Large Language Models (LLMs) are increasingly deployed in production workflows for domain-specific question answering, software development, and structured data extraction. Yet their widespread adoption has outpaced principled methods for controlling their behavior through prompting alone. In high-stakes settings, it is not enough for an LLM to be capable in aggregate; it must be robust to small input perturbations, efficient in resource usage, and aligned with explicit human and organizational preferences. This dissertation advances empirically grounded, repeatable approaches for designing and refining prompts and interaction patterns—treating prompt engineering as an engineering discipline rather than an ad hoc craft. The dissertation integrates eight tightly connected studies that collectively examine robustness under meaning-preserving input variation, prompt strategies for code generation, prompt styles for structured data generation, and systematic refinement loops that encode preferences without modifying model weights. First, using biomedical question answering as a testbed, the dissertation quantifies how semantically equivalent rephrasings can yield materially different answers, revealing brittleness to surface form and motivating robustness-aware evaluation as a prerequisite for trustworthy deployment. Next, in software engineering tasks, the dissertation demonstrates that systematic exploration of prompt variants, multi-sample generation, and quantitative benchmarking can produce LLM-generated programs that match or surpass top-rated human solutions, while clarifying when multi-agent configurations provide additional value beyond single-model prompting. The dissertation then develops a metrics-driven framework for structured data generation, showing that prompt style (e.g., JSON, YAML, hybrid CSV/prefix) induces non-trivial trade-offs among accuracy, token cost, and latency across multiple datasets and frontier models. Finally, the dissertation introduces Preference-Driven Refinement (PDR) and a critic-in-the-loop extension (PDR+Critic), formalizing iterative procedures that incorporate explicit preferences—such as quality rubrics, security constraints, and documentation standards—into successive prompts. Cross-model results show that structured refinement can improve consistency and objective quality, while also introducing overhead that must be managed through careful workflow design. Overall, this dissertation contributes experimental methodologies, benchmarks, and reusable prompting/refinement patterns that enable robust, efficient, and preference-aligned LLM behavior for specialized tasks without retraining or fine-tuning.