Search Results for "crawl site links"
Sort By:
Showing 305 open source projects for "crawl site links"
View related business solutions-
Gen AI apps are built with MongoDB AtlasMongoDB Atlas is the developer-friendly database used to build, scale, and run gen AI and LLM-powered apps—without needing a separate vector database. Atlas offers built-in vector search, global availability across 115+ regions, and flexible document modeling. Start building AI apps faster, all in one place.
-
Taking the Paper Out of WorkThe Square 9 AI-powered intelligent document processing platform takes the paper out of work and makes it easier to get things done with digital workflows.
-
1
Laravel Sitemap
Create and generate sitemaps with ease
This package can generate a sitemap without you having to add urls to it manually. This works by crawling your entire site. The generator has the ability to execute JavaScript on each page so links injected into the dom by JavaScript will be crawled as well. The easiest way is to crawl the given domain and generate a sitemap with all found links. The destination of the sitemap should be specified by $path. If you don't want a crawled link to appear in the sitemap, just don't return it in the callable you pass to hasCrawled. ... -
2
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites. -
3
crwlr
Library for Rapid (Web) Crawler and Scraper Development
This library provides kind of a framework and a lot of ready-to-use, so-called steps, that you can use as building blocks, to build your own crawlers and scrapers with. Before diving into the library, let's have a look at the terms crawling and scraping. For most real-world use cases, those two things go hand in hand, which is why this library helps with and combines both. A (web) crawler is a program that (down)loads documents and follows the links in it to load them as well. A crawler... -
4
notfoundbot
fix & archive outgoing links on your website
notfoundbot is a GitHub Action that helps you automatically maintain the correctness of your website's outgoing links. It finds links that need fixing and opens pull requests that fix them. This action is intended for websites and blogs powered by static site generators. By using post dates derived from filenames, notfoundbot searches for Wayback Machine archives of linked resources that are contemporary to the post itself: broken links in a 2011 blog post will be linked to archives from around that era. -
Employees get more done with RipplingEffortlessly manage the entire employee lifecycle, from hiring to benefits administration. Automate HR tasks, ensure compliance, and streamline approvals. Simplify IT with device management, software access, and compliance monitoring, all from one dashboard. Enjoy timely payroll, real-time financial visibility, and dynamic spend policies. Rippling empowers your business to save time, reduce costs, and enhance efficiency, allowing you to focus on growth. Experience the power of unified management with Rippling today.
-
5
Heimdall
An Application dashboard and launcher
As the name suggests Heimdall Application Dashboard is a dashboard for all your web applications. It doesn't need to be limited to applications though, you can add links to anything you like. Heimdall is an elegant solution to organize all your web applications. It’s dedicated to this purpose so you won’t lose your links in a sea of bookmarks. Why not use it as your browser start page? It even has the ability to include a search bar using either Google, Bing or DuckDuckGo. You can use the app to link to any site or application, but Foundation apps will auto-fill in the icon for the app and supply a default color for the tile. ... -
6
Nextra
Simple, powerful and flexible site generation framework
Simple, powerful, and flexible site generation framework with everything you love from Next.js. Nextra automatically converts Markdown links and images to use Next.js Link and Next.js Image when possible. No slow navigation or layout shift. -
7
Web-Check
All-in-one OSINT tool for analysing any website
...Get an insight into the inner-workings of a given website: uncover potential attack vectors, analyse server architecture, view security configurations, and learn what technologies a site is using. Currently the dashboard will show: IP info, SSL chain, DNS records, cookies, headers, domain info, search crawl rules, page map, server location, redirect ledger, open ports, traceroute, DNS security extensions, site performance, trackers, associated hostnames, carbon footprint. Stay tuned, as I'll add more soon. The aim is to help you easily understand, optimize and secure your website. -
8
Sphinx
Main repository for the Sphinx documentation builder
...It was originally created for the Python documentation, and it has excellent facilities for the documentation of software projects in a range of languages. Of course, this site is also created from reStructuredText sources using Sphinx! HTML (including Windows HTML Help), LaTeX (for printable PDF versions), ePub, Texinfo, manual pages, plain text. Semantic markup and automatic links for functions, classes, citations, glossary terms and similar pieces of information. Easy definition of a document tree, with automatic links to siblings, parents and children. ... -
9
Django jazzmin
Jazzy theme for Django
...Select2 drop-downs. Bootstrap 4 & AdminLTE UI components. You can add links to the user menu on the top right of the screen using the "usermenu_links" settings key, the format of these links is the same as with top menu, though submenus via "app" are not currently supported and will not be rendered. The side menu gets a list of all installed apps and their models that have admin classes, and creates a tree of apps and links to model admin pages. -
Kinetic Software - Epicor ERPGrow, thrive, and compete in a global marketplace with Kinetic—an industry-tailored, cognitive ERP that helps you work smarter and stay connected.
-
10
LinkChecker
Check links in web documents or full websites
LinkChecker is a free, GPL licensed website validator. LinkChecker checks links in web documents or full websites. It runs on Python 3 systems, requiring Python 3.8 or later. The version in the pip repository may be old, to find out how to get the latest code, plus platform-specific information and other advice see doc/install.txt in the source code archive. If you do not want to install any additional libraries/dependencies you can use the Docker image which is published on GitHub... -
11
Whoogle Search
A self-hosted, ad-free, privacy-respecting metasearch engine
Get Google search results, but without any ads, javascript, AMP links, cookies, or IP address tracking. Easily deployable in one click as a Docker app, and customizable with a single config file. Quick and simple to implement as a primary search engine replacement on both desktop and mobile. Autocomplete/search suggestions. POST request search and suggestion queries (when possible). View images at full res without site redirect (currently mobile only). -
12
PHPScraper
A universal web-util for PHP
PHPScraper is a universal web-scraping util for PHP, built with simplicity in mind. The goal is to make xPath Selectors optional and avoid the commonly needed boilerplate code. Just create an instance of PHPScraper, go to a website, and start collecting data. All scraping functionality can be accessed either as a function call or a property call. For example, the title can be accessed in two ways. Many common use cases are covered already. You can find prepared extractors for various HTML... -
13
fullPage.js
Create beautiful fullscreen scrolling websites fast and easy
fullPage.js is an easy-to-use library for creating beautiful, fullscreen scrolling websites/ onepage sites/ single page websites complete with all the features you need. With fullPage.js you can add landscape sliders and links to sections of your site, create smaller or bigger sections, use extensions and more! fullPage.js is compatible with all modern browsers and even some old ones like IE9 and Opera 12. It also provides touch support designed for mobile devices and touch screen computers. fullPage.js is fully supported by a great community, has good documentation and offers plenty of examples to get you started quickly. ... -
14
Infosec Reference
An Information Security Reference That Doesn't Suck
Infosec Reference is a curated knowledge base and resource repository for information security practitioners. It aggregates cheat sheets, tooling guides, protocol deep dives, incident response playbooks, and threat actor profiles—all organized under accessible categories (network, web, host, cryptography, auditing). The repo is built as a living wiki of sorts: practitioners contribute updates, expand sections, or refine explanations as the threat landscape evolves. Because security spans... -
15
SiteOne Crawler
SiteOne Crawler is a website analyzer and exporter
SiteOne Crawler is a very useful and easy-to-use tool you'll ♥ as a Dev/DevOps, website owner or consultant. Works on all popular platforms - Windows, macOS, and Linux (x64 and arm64 too). It will crawl your entire website in depth, analyze and report problems, show useful statistics and reports, generate an offline version of the website, generate sitemaps, or send reports via email. Watch a detailed video with a sample report for Astro. build website. This crawler can be used as a... -
16
SQL Explorer
Easily share data across your company via SQL queries
SQL Explorer aims to make the flow of data between people fast, simple, and confusion-free. It is a Django-based application that you can add to an existing Django site, or use as a standalone business intelligence tool. Quickly write and share SQL queries in a simple, usable SQL editor, preview the results in the browser, share links, download CSV, JSON, or Excel files (and even expose queries as API endpoints, if desired), and keep the information flowing! Comes with support for multiple connections, to many different SQL database types, a schema explorer, query history (e.g. lightweight version control), a basic security model, in-browser pivot tables, and more. ... -
17
INS
Inspiration database for Internet practitioners with no ads
INS is described as a kind of “inspiration database for internet workers” — a repository that collects and curates interesting websites, tools, links, or resources that might inspire developers, designers, or any knowledge workers. It aims to operate without ads, focusing purely on the content and resource quality, and leverages automation (e.g. GitHub Actions) to check link validity or site load speed, ensuring that listed resources remain accessible over time. For people in tech who constantly seek new tools, articles, or creative inspiration, ins serves as a living catalogue that can be browsed, contributed to, and relied upon for discovering useful or thought-provoking material without the noise of ads or clickbait. ... -
18
Minimal Mistakes Jekyll theme
Jekyll theme for building a personal site, blog, project documentation
A flexible two-column Jekyll theme. Perfect for building personal sites, blogs, and portfolios. Everything from the menus, sidebars, comments, and more can be configured or set with YAML Front Matter. Built with HTML5 + CSS3. All layouts are fully responsive with helpers to augment your content. Free to use however you want under the MIT License. Clone it, fork it, customize it, etc. Settings that affect your entire site can be changed in Jekyll’s configuration file: _config.yml, found in... -
19
DSA Bootcamp Java
This repository consists of the code samples, assignments, and notes
DSA Bootcamp Java is an open source educational repository created by Kunal Kushwaha to teach Data Structures and Algorithms (DSA) using Java. It is designed as a structured bootcamp, covering fundamental concepts to advanced problem-solving techniques. The project provides explanations, exercises, assignments, and practice problems, making it useful for both beginners and intermediate learners who want to strengthen their Java and algorithmic skills. The repository is organized into... -
20
Minima
Minima is a one-size-fits-all Jekyll theme for writers
It's Jekyll's default (and first) theme. It's what you get when you run jekyll new. Minima has been scaffolded by the jekyll new-theme command and therefore has all the necessary files and directories to have a new Jekyll site up and running with zero-configuration. From Minima v3 onwards, the base layout is named base.html instead of default.html to avoid confusing new users into assuming that name holds a special status. Users migrating from older versions with customized... -
21
jekyll-theme-8bit
A Jekyll theme inspired by classic 8bit games
A Jekyll theme inspired by classic 8-bit games. Jekyll-theme-8bit has been scaffolded by the Jekyll new-theme command and therefore has all the necessary files and directories to have a new Jekyll site up and running with zero-configuration. Snippets of code within the _includes directory that can be inserted in multiple layouts (and another include-file as well) within the same theme-gem. -
22
Smaug
Archive your Twitter/X bookmarks to markdown
...It supports configuration through JSON and can be extended or scripted as part of broader personal knowledge management systems. Because the output is plain markdown, it fits well with static site generators and digital gardens — preserving links, text, and metadata in a format that’s both platform-independent and version-friendly. -
23
glsl-sandbox
Shader editor and gallery
GLSL Sandbox is an in-browser playground for writing and sharing fragment shaders with instant visual feedback. It provides a minimal editor and a fullscreen WebGL viewport so your shader takes center stage, making it perfect for learning, live-coding, and showcasing visual experiments. The environment injects a small set of uniforms—time, resolution, mouse—so you can animate and interact without boilerplate. A public gallery lets creators browse, fork, and remix shaders, turning the site... -
24
Hugo PaperMod
A fast, clean, responsive Hugo theme
PaperMod is a fast, minimal-yet-featureful theme for the Hugo static site generator, aimed at blogs, documentation sites, and personal pages. It focuses on clean typography, responsive layouts, and sensible defaults, while exposing a large set of front-matter and config options. Built-in features include dark/light modes, archive and taxonomy views, reading-time indicators, breadcrumbs, and an optional profile/landing layout. -
25
WFDownloader App
Free batch downloader for image, wallpaper, video, audio, document,
Use as an image gallery, wallpaper, audio/music, video, document, and other media bulk downloader from supported websites. Also use to download sequential website urls that have a certain pattern (e.g. image01.png to image100.png). Also use app's built-in site crawler for advanced link search or extraction. There is also special support for forum media and open directory downloading. It's a programmable downloader and also works with password protected sites. Say goodbye to downloading one...">