Colly Files

Tons of bug fixes & little improvements & updated dependencies

What's Changed

• Added new project that uses Colly by @twiny in https://github.com/gocolly/colly/pull/525
• adding docker-slim to the list of open source projects using Colly by @kcq in https://github.com/gocolly/colly/pull/526
• update coursera example by @ysung6 in https://github.com/gocolly/colly/pull/537
• Update openedx_courses.go by @lingt-xyz in https://github.com/gocolly/colly/pull/539
• feat: support Stop() in queue by @PalanQu in https://github.com/gocolly/colly/pull/531
• Fix invalid base URL assignment by @dasshield in https://github.com/gocolly/colly/pull/549
• Add context.Context support by @WGH- in https://github.com/gocolly/colly/pull/548
• Fix handling of redirected URLs in OnResponseHeaders by @WGH- in https://github.com/gocolly/colly/pull/561
• Fix race on count fields of Collector type by @shubham14bajpai in https://github.com/gocolly/colly/pull/563
• SetProxyFunc: fix proxies rotation by @wowinter13 in https://github.com/gocolly/colly/pull/567
• Update xkcd_store.go by @BigManing in https://github.com/gocolly/colly/pull/570
• Updated xmlquery to close a known security vulnerability by @ijomeli in https://github.com/gocolly/colly/pull/582
• Fix default User-Agent when using custom headers by @WGH- in https://github.com/gocolly/colly/pull/592
• Update openedx_courses.go by @BigManing in https://github.com/gocolly/colly/pull/576
• Add httpBackend cache response callback by @slashapp in https://github.com/gocolly/colly/pull/574
• Update README.md by @seversky in https://github.com/gocolly/colly/pull/595
• Change the type assersion for Attr by @moritamori in https://github.com/gocolly/colly/pull/588
• Update User-Agent strings for Desktop by @guessi in https://github.com/gocolly/colly/pull/605
• add quote crawler using colly by @eval-exec in https://github.com/gocolly/colly/pull/603
• Fix golint issues by @waltton in https://github.com/gocolly/colly/pull/620
• Replace Travis with GitHub Actions by @alexhraber in https://github.com/gocolly/colly/pull/669
• Use github.com/nlnwa/whatwg-url for URL parsing by @WGH- in https://github.com/gocolly/colly/pull/673
• Fix Async ignoring its arguments by @ljbink in https://github.com/gocolly/colly/pull/652
• fix: generated code has compile error by @darjun in https://github.com/gocolly/colly/pull/629
• moving filterings to one function (checkFilters) and use it in case of redirect too by @joelazar in https://github.com/gocolly/colly/pull/638
• Fix Contributing.md link by @SaladinoBelisario in https://github.com/gocolly/colly/pull/641
• Work around invalid URL encoding in path by @WGH- in https://github.com/gocolly/colly/pull/675
• Host & Custom Header Support by @ErikOwen in https://github.com/gocolly/colly/pull/691
• Code cleanup by @WGH- in https://github.com/gocolly/colly/pull/678
• Add GitHub issue templates and nudge questions to Stack Overflow by @WGH- in https://github.com/gocolly/colly/pull/702
• Fix redirects ignoring AllowURLRevisit=false by @WGH- in https://github.com/gocolly/colly/pull/681
• Bump whatwg-url version by @WGH- in https://github.com/gocolly/colly/pull/713
• fix some typos by @cuishuang in https://github.com/gocolly/colly/pull/736
• feat: Add MaxRequests paramete by @guilhem in https://github.com/gocolly/colly/pull/722
• Update CI configuration by @WGH- in https://github.com/gocolly/colly/pull/749
• upgrade Cascadia version by @kinoute in https://github.com/gocolly/colly/pull/650
• Bump golang.org/x/net from 0.0.0-20220114011407-0dd24b26b47d to 0.7.0 by @dependabot in https://github.com/gocolly/colly/pull/756
• Support websites redirecting to the same page when AllowURLRevisit is disabled by @WGH- in https://github.com/gocolly/colly/pull/763
• Update User-Agent strings for 2023 by @guessi in https://github.com/gocolly/colly/pull/765
• Fix data races by @WGH- in https://github.com/gocolly/colly/pull/771
• Replaced deprecated io/ioutil package with proper implementations of … by @zachary-walters in https://github.com/gocolly/colly/pull/785
• Bump golang.org/x/net from 0.7.0 to 0.17.0 by @dependabot in https://github.com/gocolly/colly/pull/789
• put accept header before OnRequest by @k4lizen in https://github.com/gocolly/colly/pull/786
• Fix Bug: retry scrape will lost POST requestData by @Shinku-Chen in https://github.com/gocolly/colly/pull/794
• scrapper -> scraper Update README.md by @ideabrian in https://github.com/gocolly/colly/pull/809
• Test on Go 1.22, drop old versions by @WGH- in https://github.com/gocolly/colly/pull/810
• Bump google.golang.org/protobuf from 1.24.0 to 1.33.0 by @dependabot in https://github.com/gocolly/colly/pull/807
• Implement content sniffing for HTML parsing by @WGH- in https://github.com/gocolly/colly/pull/808
• Add headers/ua to robots.txt request by @makew0rld in https://github.com/gocolly/colly/pull/838
• doc: mention that OnScraped executes after OnHTML and OnXML by @aladmit in https://github.com/gocolly/colly/pull/832

New Contributors

• @twiny made their first contribution in https://github.com/gocolly/colly/pull/525
• @kcq made their first contribution in https://github.com/gocolly/colly/pull/526
• @ysung6 made their first contribution in https://github.com/gocolly/colly/pull/537
• @lingt-xyz made their first contribution in https://github.com/gocolly/colly/pull/539
• @PalanQu made their first contribution in https://github.com/gocolly/colly/pull/531
• @dasshield made their first contribution in https://github.com/gocolly/colly/pull/549
• @shubham14bajpai made their first contribution in https://github.com/gocolly/colly/pull/563
• @wowinter13 made their first contribution in https://github.com/gocolly/colly/pull/567
• @BigManing made their first contribution in https://github.com/gocolly/colly/pull/570
• @ijomeli made their first contribution in https://github.com/gocolly/colly/pull/582
• @slashapp made their first contribution in https://github.com/gocolly/colly/pull/574
• @seversky made their first contribution in https://github.com/gocolly/colly/pull/595
• @moritamori made their first contribution in https://github.com/gocolly/colly/pull/588
• @guessi made their first contribution in https://github.com/gocolly/colly/pull/605
• @eval-exec made their first contribution in https://github.com/gocolly/colly/pull/603
• @waltton made their first contribution in https://github.com/gocolly/colly/pull/620
• @alexhraber made their first contribution in https://github.com/gocolly/colly/pull/669
• @ljbink made their first contribution in https://github.com/gocolly/colly/pull/652
• @darjun made their first contribution in https://github.com/gocolly/colly/pull/629
• @joelazar made their first contribution in https://github.com/gocolly/colly/pull/638
• @SaladinoBelisario made their first contribution in https://github.com/gocolly/colly/pull/641
• @ErikOwen made their first contribution in https://github.com/gocolly/colly/pull/691
• @cuishuang made their first contribution in https://github.com/gocolly/colly/pull/736
• @guilhem made their first contribution in https://github.com/gocolly/colly/pull/722
• @kinoute made their first contribution in https://github.com/gocolly/colly/pull/650
• @dependabot made their first contribution in https://github.com/gocolly/colly/pull/756
• @zachary-walters made their first contribution in https://github.com/gocolly/colly/pull/785
• @k4lizen made their first contribution in https://github.com/gocolly/colly/pull/786
• @Shinku-Chen made their first contribution in https://github.com/gocolly/colly/pull/794
• @ideabrian made their first contribution in https://github.com/gocolly/colly/pull/809
• @makew0rld made their first contribution in https://github.com/gocolly/colly/pull/838
• @aladmit made their first contribution in https://github.com/gocolly/colly/pull/832

Full Changelog: https://github.com/gocolly/colly/compare/v2.1.0...v2.2.0