[go: up one dir, main page]

Add SWE Bench seeder for DAP evaluations

What does this MR do and why?

Add rake task to seed swebench data hosted on langsmith.

  • Add rake task gitlab:duo:swe_bench_seeder
  • Fetch examples from LangSmith dataset (configurable via env vars).
  • Clone repositories from GitHub and create issues from problem statements
  • Create an output dataset on langsmith with a single column containing the list of created issues to be used by cef.

References

Seeded projects Seeded issue
image image

How to set up and validate locally

Example running the full seeder:

bundle exec rake gitlab:duo:swe_bench_seeder

Example with a project:

bundle exec rake 'gitlab:duo:swe_bench_seeder[matplotlib]'

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Bruno Cardoso

Merge request reports

Loading