Add SWE Bench seeder for DAP evaluations
What does this MR do and why?
Add rake task to seed swebench data hosted on langsmith.
- Add rake task
gitlab:duo:swe_bench_seeder - Fetch examples from LangSmith dataset (configurable via env vars).
- Clone repositories from GitHub and create issues from problem statements
- Create an output dataset on langsmith with a single column containing the list of created issues to be used by
cef.
References
| Seeded projects | Seeded issue |
|---|---|
|
|
How to set up and validate locally
Example running the full seeder:
bundle exec rake gitlab:duo:swe_bench_seeder
Example with a project:
bundle exec rake 'gitlab:duo:swe_bench_seeder[matplotlib]'
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Bruno Cardoso

