Why ROUGE Lies About Your LLM Outlines and How Grounded Fine-Tuning Fixes It

[roshankumar]

Key Takeaways​

• The outline should be handled as structured model output: a title paired with paragraph IDs, rather than as text. That small shift makes it machine-checkable for grounding.
• The hybrid labeling should be compared to an extractive-only and to a generative-only labeling on the same held-out data set when possible; the current version of Table 3 summarizes evidence from the literature rather than gives us results of this comparison.
• Noisy span boundaries require data-quality checks; without the audit-trail experiment, comparisons of model size should be treated as hypotheses rather than facts.
• Evaluate in layers: The structural and basic analysis, rubric, and human review. We will do comparisons using ROUGE and BLEU against a single human reference, which is to put forth penalties to the correct paraphrase, which in turn will rank the faulty in terms of structure.
• Release a measure like (for example, share of documents with acceptable outlines) and report next to the results the human-audited share. Average token overlap is not a release metric.

Executive Summary​

Creating a draft for an extensive unstructured document might initially seem like just summarizing the primary points as a very difficult task. However, it is essentially a structured prediction problem. Each of the elements should refer to specific sections of the text; the draft should contain everything there is in the documents with no information missing or repeated; and the headings should be short, similar in structure and content. In this paper, we will describe a supervised fine-tuning (SFT) approach which uses paragraph IDs as a common basis for tying labels, model output, and assessment. We generate training labels based on extractive candidates and generative rewording, we reject those that do not meet relevance requirements prior to initiating the training stage, and we utilize various assessors connecting evaluation results with acquisition by the reader. Also, we present cases where n-gram metrics fail us and report on the use of prompted models.

Introduction​

Most teams start by prompting. Asking a general-purpose model to "generate a table of contents" produces plausible results quickly, but three problems grow with scale. Titles mention content the section does not contain. Sections go out of order or repeat. The output format varies between each run. We see this type of issue often reported out there [1] also at NIST they put forth confabulation as a unique generative AI risk [2]. Fine-tuning is what usually follows. SFT of curated sets of examples did, in fact, change model action[3, 4]; also, we see small sets of high-quality data may be enough[5]. Training is typically the issue. The hard part is in defining the target and the evaluator, which in turn produces a better outcome. This article looks at those two elements and puts forth a reproducible SFT pipeline for structured outline generation.

Background​

Extractive versus abstractive generation. The extractive method does well in accuracy but at times reports odd results. See et al. showed that a pointer-generator model, which can copy from the source as well as generate, reduced factual errors and repetition compared with a purely abstractive baseline on news summarization [6]. We apply the same idea to label construction rather than to the model architecture.

Segmentation. Deciding where sections begin can be learned with supervision [7]. In one part of an outline it is possible to use one title, whilst in an adjacent segment a completely different title is necessary.

Attribution. The AIS framework [8] and citation benchmarks [9] turn "is this supported by an identified source?" into a question that can be answered. Paragraph IDs make the source explicit for every outline entry.

Problem Analysis​

A production outline generator has to work within the confines of what a single quality metric includes.

Constraint	What failure looks like	Why prompting alone struggles
Coverage	Paragraphs that belong to no section	The model has no explicit notion of the paragraph inventory
Monotonic, non-overlapping spans	Section 3 starts before section 2 ends	Cross-entry ordering is not enforced by free-text decoding
Faithful titles	A title names a topic the span never discusses	Abstractive phrasing drifts toward plausible content
Consistent granularity	Twenty one-paragraph sections, or one giant section	Granularity is a product decision, not a language skill
Long inputs	Middle sections get weaker titles	Models use mid-context information less reliably [10]

Table 1: Constraints a production outline generator must satisfy.

The most critical aspect is the final line: a methodology dependent on the model's recognition of structure fails in the same lengthy documents where outlines are most beneficial.

Architecture and Design​

Figure 1 presents our pipeline. At its core is the output contract. Instead of free-form headings, it has a JSON array of {title, startid, endid, level} which point to paragraphs in the source text.



Figure 1: Grounded SFT pipeline. Paragraphs which have ID’s created at the time of segmentation carry those IDs through to the

Stable paragraph IDs. Every paragraph is assigned an ID, for example, [p17], which does not change with any version of a document. Therefore, the ID represents each label; it predicts and finally gives the evaluator verdict about a certain entity.

Hybrid label builder. The section boundaries need attention first. When a paper includes headings, formatting features or numbered sections of text, they will already set them. A model in this case starts specifying the boundaries as paragraph-ID intervals. This will be done under the strictest filtering and heavy spot-checking. A title for each section will be a part of either the present heading or the best lead sentence. There is a generative pass where the draft titles will be converted into short phrases that are going to be saying the same but with different words. The extractive pass defines what the sections are, while the generative pass determines what they will look like.

One caution: The use of a model in the teacher’s rewrites communicates the teacher’s wording and blind spots directly to the student in the learning process. In order to eliminate the occurrence of such interference in the evaluating process, rotating rewrites and keeping a single human-written instance in the sessions should be done.

Grounding filter. A determinative filter first eliminates titles that contain spans that are absent, overlapping, unordered, or out-of-range, implementing the same validator as mentioned below. In the next step, an entailment verification eliminates titles whose paragraphs do not support their content. Also, it is noted that models that perform entailment better do so as reported by human assessors’ reports of content fidelity.

Turning a title into a hypothesis. “Overview” is only a title, not a claim. An NLI model therefore cannot process it as-is. A title must outperform its own component words in terms of exact match in the full title. For vague titles that pass both checks, they are sent to the rubric judge for assessment of how specific they are.

Constrained decoding, and why we still validate. Grammar-constrained decoding in [24, 25] also means that the output parses and that each field has the right type, and we use it at serving time. We did not find in the JSON Schema engines that we looked at rules like “each entry starts where the previous one ended” or “children stay inside their parent”. For those, we would require a custom decoding-time state machine, which in turn would have to be redeveloped for each serving stack. Also, we found that the validator is a requirement which we used to filter training labels and to score outputs offline. We parse the syntax against the schema; the validator alone determines whether the structure is valid.

Training. In practice, fine-tuning usually follows the standard self-supervised framework, calculating loss only for the goal tokens. The splits of train and test sets happen at the source level, thus making it impossible for leakage of nearly identical templates from a certain source into the test set. Both full fine-tuning and LoRA can be utilized. LoRA only trains a small subset of weights, which allows the retraining of the model after fixing the labels to be performed in a cost- and time-effective manner. The framework utilized can be either a PyTorch- or JAX-based stack like AXLearn.

Implementation​

A training example is deliberately simple:

INPUT:
[p1] Overview of the onboarding process ...
[p2] New accounts are created by ...
...
[p24] Frequently asked questions ...
 
TARGET:
[{"title": "Onboarding overview", "start_id": 1, "end_id": 3, "level": 1},
 {"title": "Creating an account", "start_id": 4, "end_id": 9, "level": 1},
 ...]

The structural validator below runs in three places: in the grounding filter on labels, as a hard gate on model output at serving time, and as the first evaluator layer.

def validate_outline(entries, n_paragraphs, max_depth=3, max_title_words=12):
    errors = []
    stack = [] # open entries: [index, start, end, next_child_start]
    next_top = 1 # required start of next level-1 entry; None = resync
 
    def close_to(depth):
        while len(stack) > depth:
            idx, ps, pt, nxt = stack.pop()
            # children exist but stop short of the parent's end
            if nxt not in (ps, pt + 1, None):
                errors.append((idx, "children_incomplete"))
 
    for i, e in enumerate(entries):
        s, t, lvl = e["start_id"], e["end_id"], e["level"]
        title = e["title"].strip()
        if not title or len(title.split()) > max_title_words:
            errors.append((i, "bad_title_length"))
        max_lvl = min(max_depth, len(stack) + 1)
        if not isinstance(lvl, int) or not 1 <= lvl <= max_lvl:
            errors.append((i, "bad_level"))
            continue # a bad entry never becomes a parent
        if not (1 <= s <= t <= n_paragraphs):
            errors.append((i, "span_out_of_range"))
            close_to(lvl - 1)
            if lvl == 1:
                next_top = None
            else:
                stack[-1][3] = None
            continue # a bad span never moves the cursor
        close_to(lvl - 1)
        if lvl == 1:
            if next_top is not None and s != next_top:
                errors.append((i, "gap_or_overlap"))
            next_top = t + 1
        else:
            pidx, ps, pt, nxt = stack[-1]
            if not (ps <= s <= t <= pt):
                errors.append((i, "child_outside_parent"))
            elif nxt is not None and s != nxt:
                errors.append((i, "gap_or_overlap"))
            stack[-1][3] = t + 1
        stack.append([i, s, t, s])
    close_to(0)
    if next_top != n_paragraphs + 1:
        errors.append((-1, "incomplete_coverage"))
    return errors

In Level-1 we see that entries which in turn also begin where the previous left off and thus between them full coverage of each paragraph is achieved. Nested entries title their parent the same way: the first child starts at the parent's first paragraph, siblings are contiguous, and the last child ends at the parent's last paragraph. Levels cannot be skipped, and level 0 or anything deeper than max_depth is rejected.Entries that are invalid or outside the allowed range will be reported once, then ignored. They will neither produce children nor shift the intended start, and a single invalid entry will not trigger a chain of erroneous vapor overlap errors.



Instead of providing a boolean response, the function gives back its coded indications of the system error. These codes present the structure of errors in the rectification process. We see that mostly what happens is gap or overlap, also what we have is that which is of low quality does not present.



At the same time the same contract applies. When values do not pass validation we go through the process again once. If a retry fails we present an extractive outline. An extractive outline uses the sections from the document and headers if present which at that point pass the validator. In the case the document does not have headers which is a large in most cases determination by the model we use a simplified lexical cohesion segmentation approach that is similar to TextTiling [26] with what we term lead sentence headers. We present this as a weak solution which we separately report the results for headerless fall back. For the great majority of readers a structurally valid out line is more useful than a fluent out line which has wrong spans and the fall back sets a floor for the fine tuned model.

Evaluation: Why ROUGE Is Not Enough​

ROUGE and BLEU measure n-gram overlap with a reference [16, 17], and for outlines they fail in both directions: a correct but paraphrased title is penalized, and a structurally broken outline can still score well. Onboarding overview of what we do in that process of getting started in the platform “Creating out accounts” “We will be verifying your details once you begin the process of account creation which may include a series of questions to confirm your identity for security purposes” also see how we set up your billing structure which includes various plans to fit your specific needs “How to invite your colleagues to start using the product with you in their workflows” .



Also we have the fine print that sets the rules for role assignment and what it means for the people you bring in as part of your team “And finally what happens after you finish setting things up and beginning your journey here” Candidate A has just taken the first 4 topics straight from what is provided without change, put in a different order then has combined our 5th and 6th into a single step, then has omitted the last one completely so that we as authors had no say in what is included or excluded in that final 1/3 of the onboarding process. Candidate B did an excellent job at rewriting the actual titles used and also followed the authors structure and order exactly as presented. Also note that which paragraphs in the original were not covered, that is also what is not present in the write ups by candidate B.

Candidate	ROUGE-1 F1	ROUGE-L F1	Passes a structural validator?
A (copied titles, merged section, missing conclusion)	0.92	0.86	No (incomplete_coverage)
B (paraphrased titles, exact tiling)	0.14	0.14	Yes

Table 2: ROUGE evaluated over the joined and lowered versions of each candidate’s title against the reference (example put forth rather than from production data). For overlap measures which goes to A; a human reader and evaluator below go to B.



The evaluator stack therefore has layers, ordered from cheapest and most deterministic to most expensive (Figure 2):

1. Structural validator. The code above, used as a hard gate. An outline that fails it is not "partially good".
2. Grounding check. Each of the titles is evaluated against its anchored paragraphs with the NLI procedure we presented before.
3. Rubric judge. Using a fixed rubric, an LLM judge rates the response for granularity, parallelism, and usefulness. LLM judges can align well with human ratings [19, 20], but they are sensitive to answer position and verbosity [20, 21]. The rubric goes through each outline separately instead of putting them in pairs, also the judge’s prompt and model are updated for each version. Because the teacher wrote the generative titles in the training labels, the judge comes from a different model family than the teacher; otherwise it may favour teacher-style phrasing in the same way the student imitates it.
4. Human audit. People rate a sample every release, to calibrate the judge and to detect drift.

Results​

This ‘release metric’ is in a product shape, what we may call a good outline rate which is the count of held out documents that pass all the auto layers and meet the auto rubric threshold. Per document -- all or nothing this metric brings to light what other average per entry scores may hide. If one out of twenty spans fails which in turn ruins the purpose for that particular reader; but if you average across entries that one failure brings in almost no notice at all. That is, it is this multi-metric argument applied to a single product [22].



Since at present there is no independent verifiable same corpus outline benchmark, Table 3 reports the relevant peer reviewed public data instead. The results stated do relate to different tasks, meaning that they cannot be considered as an assessment of the proposed system. Future head-to-head evaluations will involve prompted generation, extractive-only SFT, generative SFT, and hybrid label SFT, carrying out evaluations using the same source distinct splits, validation rules, and human audit.

Published source	Original research task	Reported public evidence	Relevance to proposed pipeline	Interpretation / limitation
Koshorek et al. (NAACL 2018) [7]	Supervised text segmentation	Introduces a large Wikipedia-derived labeled segmentation dataset and demonstrates generalization to unseen natural text.	Public basis for learning contiguous document boundaries.	Not an evaluation of generated outline titles or hybrid SFT.
Liu et al. (EMNLP 2023) [19]	Human-aligned NLG evaluation	G-Eval reports Spearman correlation 0.514 with human judgments on a summarization task.	Supports using a rubric-based evaluator calibrated against human ratings.	Correlation is not an outline acceptance rate; different task and benchmark.
Huang et al. (2021), GovReport [27]	Long-document summarization	Public corpus of government reports with long documents and reference summaries; question-summary hierarchies are also available.	Potential public source corpus for document-structure evaluation.	No hybrid-SFT outline performance or same-set ablation reported.
Proposed hybrid outline SFT (this article)	Paragraph-ID grounded outline generation	No publicly verified comparative outcome reported in this manuscript.	Specified method and validation protocol; prospective evaluation.	Automatic pass, human audit, fallback, sample size and 95% CIs require new measurements.

Table 3: Publicly reported evidence relevant to outline segmentation and evaluation. Results are reproduced from the cited publications on their original tasks; they are NOT a held-out comparison of the proposed hybrid SFT pipeline. No original system-level acceptance rate, fallback rate, or confidence interval is claimed.



Supervised document division and the final product are done by a human. We do not see that hybrid-label SFT is better than prompted SFT, extractive-only, and generative-only approaches that use the same dataset. What is needed is a controlled evaluation against a common test corpus which includes human audits and bootstrapping methods before we can say that [7, 19].

Performance and Scalability​

Cost is what determines the price. Through the use of paragraph IDs, which is what makes chunking a better option, we are able to break up long texts into different sets of paragraphs in various overlapping windows, which in turn also we see to do a great job in terms of mid-context performance. The issue we are presented with is that of putting these windows back together.

• Duplicate boundary sections. A certain part of the overlapping area occurs in both of the windows. We use the text from the window in which the part is farther away from the edge of the window because that window has had more context.
• Level consistency. Every window independently assigns levels. First, we find the overlapping sections between the two windows and use the matching between the second window's levels and the first one's. The other segments in the second window are later adjusted by the same offset as the corresponding segments.
• Titles: Depending upon the requirements of the project, there might still be differences in the content although the sections might run parallel to each other.
• Revalidation. The structural validator is used for the connected outline, which is put through it again and again. Merge bugs can be found as gaps or overlaps at the edges of windows, which makes them apparent.

The structural validator is at no cost and goes first, also expensive judge calls are for valid outputs.

Security and Risk Considerations​

The documents inputted are considered insecure, and this increases the chances of manipulation. Only the data that is validated is a title and an integer figure. We put in place a process to screen out training documents that contain sensitive issues, as once it is fine-tuned, we may be able to recover the text used for training. In the “Map”, “Measure” and “Manage” stages of the NIST AI RMF, we document these issues as well as the results of the conducted evaluations.

Operational Excellence​

We log into our system every prediction that is put out, as well as the model it is based on, the prompt we used and the validation performance we get.

Trade-offs and Limitations​

• Paragraph granularity is a ceiling. Boundaries in the middle of a paragraph do not present well, which is usual for prose but not for dense technical text.
• Hybrid labeling adds pipeline complexity. A segmenter, classifier, tagger, and filter-out can each drift. A segmenter, classifier, tagger, and filter-out can each drift.
• Judge dependence. The judge that we use in rating is a model which has known biases [21]. We don’t have a human audit of the release metric, which may thus end up favoring the judge’s agenda rather than the reader’s.
• When not to fine-tune. In cases of a small document corpus, uniform or high document turnover, using the same prompter behind the same validator and fallback may be enough at a lower operational cost. A comparison against the prompting model as the baseline in a head-to-head setting with the same source-disjoint held-out set is the comparison to conduct before proceeding with SFT.

Production Lessons Learned​

Lesson 1: Fix labels before tuning hyperparameters. When a label falls within the correct span but starts or ends in the wrong place, the model can learn that loose boundaries are acceptable and repeat the mistake. The proposed remedy is to correct those spans before measuring model size; as a result, no controlled model-size ablations are currently reported. Typed validator errors help diagnose the issue. If most withheld errors are vapor overlap, the labels should be examined before changing the model. This is why Figure 1 places the grounding filter before the SFT dataset.

Lesson 2: Design the fallback first. When the fine-tuned model fails a deterministic extractive fallback which in turn puts the system at a safe state, also sets a base performance that the model must outperform.

Best Practices​

• Each source item will be given an ID. We maintain a consistent schema and validator which is used throughout the label, train, deploy, and evaluate phases.
• Before we enter the training stage, hybrid labels will be verified against whichever is the source they came from. Each set of labels must be looked at separately.
• The evaluators should be classified according to their type, from deterministic to model-based, and the less expensive ones should be filtered out.
• Jury prompts and models should be updated without causing any confusion regarding the jury's affiliation with the teaching model.
• Finally, the metric of every automation release should be defined with the findings of the human feedback taken into account.

Future Outlook​

Having longer context windows will eliminate the necessity of chunking naturally, but it will not eliminate the need for grounding in a direct way [10]. A continuing development is that of attribution as a first-class output: models that generate segments or references by conception and are assessed with attribution metrics, not based on overlap [8, 9].

Conclusion​

Reliable outlines are a result of the model’s structure, which in turn is a result of its architecture rather than the model’s size. Use the same ID for each paragraph in every outline you create and have every outline element reference those IDs. Create labels that are extractive and very ground in the text and for the title which is to be generated, use a clean, easy-to-read format; also do a pre-training filter. Then evaluate the outlines in stages from a computer-based validator to a human review, which should mean that the better the score, the better the outline for the reader.

References​

[1] Z. Ji, N. Lee, R. Frieske, et al., "Survey of Hallucination in Natural Language Generation," ACM Computing Surveys, vol. 55, no. 12, 2023. [Online]. Available: https://arxiv.org/abs/2202.03629

[2] National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile," NIST AI 600-1, Jul. 2024. [Online]. Available: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

[3] J. Wei, M. Bosma, V. Y. Zhao, et al., "Finetuned Language Models Are Zero-Shot Learners," in Proc. Int. Conf. Learning Representations (ICLR), 2022. [Online]. Available: https://arxiv.org/abs/2109.01652

[4] L. Ouyang, J. Wu, X. Jiang, et al., "Training language models to follow instructions with human feedback," in Advances in Neural Information Processing Systems 35 (NeurIPS), 2022. [Online]. Available: https://arxiv.org/abs/2203.02155

[5] C. Zhou, P. Liu, P. Xu, et al., "LIMA: Less Is More for Alignment," in Advances in Neural Information Processing Systems 36 (NeurIPS), 2023. [Online]. Available: https://arxiv.org/abs/2305.11206

[6] A. See, P. J. Liu, and C. D. Manning, "Get To The Point: Summarization with Pointer-Generator Networks," in Proc. 55th Annual Meeting of the ACL, 2017, pp. 1073–1083. [Online]. Available: https://aclanthology.org/P17-1099/

[7] O. Koshorek, A. Cohen, N. Mor, M. Rotman, and J. Berant, "Text Segmentation as a Supervised Learning Task," in Proc. NAACL-HLT, 2018, pp. 469–473. [Online]. Available: https://aclanthology.org/N18-2075/

[8] H. Rashkin, V. Nikolaev, M. Lamm, et al., "Measuring Attribution in Natural Language Generation Models," Computational Linguistics, vol. 49, no. 4, pp. 777–840, 2023. [Online]. Available: https://aclanthology.org/2023.cl-4.2/

[9] T. Gao, H. Yen, J. Yu, and D. Chen, "Enabling Large Language Models to Generate Text with Citations," in Proc. EMNLP, 2023, pp. 6465–6488. [Online]. Available: https://aclanthology.org/2023.emnlp-main.398/

[10] N. F. Liu, K. Lin, J. Hewitt, et al., "Lost in the Middle: How Language Models Use Long Contexts," Transactions of the ACL, vol. 12, pp. 157–173, 2024. [Online]. Available: https://aclanthology.org/2024.tacl-1.9/

[11] J. Maynez, S. Narayan, B. Bohnet, and R. McDonald, "On Faithfulness and Factuality in Abstractive Summarization," in Proc. 58th Annual Meeting of the ACL, 2020, pp. 1906–1919. [Online]. Available: https://aclanthology.org/2020.acl-main.173/

[12] W. Kryściński, B. McCann, C. Xiong, and R. Socher, "Evaluating the Factual Consistency of Abstractive Text Summarization," in Proc. EMNLP, 2020, pp. 9332–9346. [Online]. Available: https://aclanthology.org/2020.emnlp-main.750/

[13] E. J. Hu, Y. Shen, P. Wallis, et al., "LoRA: Low-Rank Adaptation of Large Language Models," in Proc. ICLR, 2022. [Online]. Available: https://arxiv.org/abs/2106.09685

[14] A. Paszke, S. Gross, F. Massa, et al., "PyTorch: An Imperative Style, High-Performance Deep Learning Library," in Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. [Online]. Available: https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html

[15] M. Lee, C. Lan, T. Gunter, et al., "AXLearn: Modular, Hardware-Agnostic Large Model Training," arXiv:2507.05411, 2025. [Online]. Available: https://arxiv.org/abs/2507.05411

[16] C.-Y. Lin, "ROUGE: A Package for Automatic Evaluation of Summaries," in Text Summarization Branches Out (ACL Workshop), 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013/

[17] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "BLEU: a Method for Automatic Evaluation of Machine Translation," in Proc. 40th Annual Meeting of the ACL, 2002, pp. 311–318. [Online]. Available: https://aclanthology.org/P02-1040/

[18] J. Novikova, O. Dušek, A. Cercas Curry, and V. Rieser, "Why We Need New Evaluation Metrics for NLG," in Proc. EMNLP, 2017, pp. 2241–2252. [Online]. Available: https://aclanthology.org/D17-1238/

[19] Y. Liu, D. Iter, Y. Xu, et al., "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment," in Proc. EMNLP, 2023, pp. 2511–2522. [Online]. Available: https://aclanthology.org/2023.emnlp-main.153/

[20] L. Zheng, W.-L. Chiang, Y. Sheng, et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," in NeurIPS Datasets and Benchmarks Track, 2023. [Online]. Available: https://arxiv.org/abs/2306.05685

[21] P. Wang, L. Li, L. Chen, et al., "Large Language Models are not Fair Evaluators," in Proc. 62nd Annual Meeting of the ACL, 2024, pp. 9440–9450. [Online]. Available: https://aclanthology.org/2024.acl-long.511/

[22] P. Liang, R. Bommasani, T. Lee, et al., "Holistic Evaluation of Language Models," Transactions on Machine Learning Research, 2023. [Online]. Available: https://arxiv.org/abs/2211.09110

[23] National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)," NIST AI 100-1, Jan. 2023. [Online]. Available: https://www.nist.gov/itl/ai-risk-management-framework

[24] B. T. Willard and R. Louf, "Efficient Guided Generation for Large Language Models," arXiv:2307.09702, 2023. [Online]. Available: https://arxiv.org/abs/2307.09702

[25] S. Geng, M. Josifoski, M. Peyrard, and R. West, "Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning," in Proc. EMNLP, 2023. [Online]. Available: https://aclanthology.org/2023.emnlp-main.674/

[26] M. A. Hearst, "TextTiling: Segmenting Text into Multi-paragraph Subtopic Passages," Computational Linguistics, vol. 23, no. 1, pp. 33–64, 1997. [Online]. Available: https://aclanthology.org/J97-1003/

[27] L. Huang et al., “Efficient Attentions for Long Document Summarization,” 2021. GovReport dataset and documentation: https://gov-report-data.github.io/.

About the Author​

Roshan Kumar is a Senior Machine Learning Engineer with more than ten years of software engineering experience. His current work focuses on large language model fine-tuning, structured and grounded generation, prompt optimization and automated evaluation for search-facing applications. Before that, he spent five years building event-driven serverless backends and microservices for large-scale retail inventory replenishment and purchase-order management. He holds an MS in Computer Science from New York University.