LLM-Linked Chatbot Platforms for Seeded Clinical Trial Randomization Workflows: A Benchmarking Study of Reproducibility, Allocation Integrity, and Operational Traceability

Randomization sequence generation is essential in randomized controlled trials, but access to trial-management systems or statistical support may be limited in some settings. This in silico technical benchmarking study evaluated whether four LLM-linked chatbot interfaces can faithfully execute pre-specified deterministic Python code to generate a randomized sequence under fixed-seed conditions. In Experiment 1, four investigators performed 1200 fixed-seed Python runs across two sample-size scenarios (n = 30 and n = 50), benchmarked against seeded Excel/VBA and R Console workflows.

In Experiment 2, the same investigators performed 320 NL-only runs without code submission or seed specification. A supplementary permuted block benchmark (n = 60; blocks of six) added 640 runs across both prompting conditions. Fixed-seed code execution achieved 100% exact reproducibility, allocation integrity, format compliance, and operational completion across all platforms.

NL-only prompting preserved allocation integrity, format compliance, and operational completion (100%) but yielded 0% exact reproducibility in both simple and permuted block randomization. These findings support only a constrained interpretation: chatbot-mediated reproducibility depends on executable code, fixed-seed specification, preserved documentation, and human verification. These interfaces should not replace dedicated randomization software or validated trial-management systems.