Search Results for "crawl site links"
Sort By:
Showing 236 open source projects for "crawl site links"
View related business solutions-
Gen AI apps are built with MongoDB AtlasMongoDB Atlas is the developer-friendly database used to build, scale, and run gen AI and LLM-powered apps—without needing a separate vector database. Atlas offers built-in vector search, global availability across 115+ regions, and flexible document modeling. Start building AI apps faster, all in one place.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
Laravel Sitemap
Create and generate sitemaps with ease
This package can generate a sitemap without you having to add urls to it manually. This works by crawling your entire site. The generator has the ability to execute JavaScript on each page so links injected into the dom by JavaScript will be crawled as well. The easiest way is to crawl the given domain and generate a sitemap with all found links. The destination of the sitemap should be specified by $path. If you don't want a crawled link to appear in the sitemap, just don't return... -
2
Sphinx
Main repository for the Sphinx documentation builder
Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation, and it has excellent facilities for the documentation of software projects in a range of languages. Of course, this site is also created from reStructuredText sources using Sphinx! HTML (including Windows HTML Help), LaTeX (for printable PDF versions), ePub, Texinfo, manual pages, plain text... -
3
Whoogle Search
A self-hosted, ad-free, privacy-respecting metasearch engine
Get Google search results, but without any ads, javascript, AMP links, cookies, or IP address tracking. Easily deployable in one click as a Docker app, and customizable with a single config file. Quick and simple to implement as a primary search engine replacement on both desktop and mobile. Autocomplete/search suggestions. POST request search and suggestion queries (when possible). View images at full res without site redirect (currently mobile only). Light/Dark/System theme modes... -
4
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites. -
Simple, Secure Domain RegistrationRegister or renew your domain and pay only what we pay. No markups, hidden fees, or surprise add-ons. Choose from over 400 TLDs (.com, .ai, .dev). Every domain is integrated with Cloudflare's industry-leading DNS, CDN, and free SSL to make your site faster and more secure. Simple, secure, at-cost domain registration.
-
5
Web-Check
All-in-one OSINT tool for analysing any website
Comprehensive, on-demand open source intelligence for any website. Get an insight into the inner-workings of a given website: uncover potential attack vectors, analyse server architecture, view security configurations, and learn what technologies a site is using. Currently the dashboard will show: IP info, SSL chain, DNS records, cookies, headers, domain info, search crawl rules, page map, server location, redirect ledger, open ports, traceroute, DNS security extensions, site performance... -
6
SiteOne Crawler
SiteOne Crawler is a website analyzer and exporter
SiteOne Crawler is a very useful and easy-to-use tool you'll ♥ as a Dev/DevOps, website owner or consultant. Works on all popular platforms - Windows, macOS, and Linux (x64 and arm64 too). It will crawl your entire website in depth, analyze and report problems, show useful statistics and reports, generate an offline version of the website, generate sitemaps, or send reports via email. Watch a detailed video with a sample report for Astro. build website. This crawler can be used as a command... -
7
SQL Explorer
Easily share data across your company via SQL queries
SQL Explorer aims to make the flow of data between people fast, simple, and confusion-free. It is a Django-based application that you can add to an existing Django site, or use as a standalone business intelligence tool. Quickly write and share SQL queries in a simple, usable SQL editor, preview the results in the browser, share links, download CSV, JSON, or Excel files (and even expose queries as API endpoints, if desired), and keep the information flowing! Comes with support for multiple... -
8
glsl-sandbox
Shader editor and gallery
GLSL Sandbox is an in-browser playground for writing and sharing fragment shaders with instant visual feedback. It provides a minimal editor and a fullscreen WebGL viewport so your shader takes center stage, making it perfect for learning, live-coding, and showcasing visual experiments. The environment injects a small set of uniforms—time, resolution, mouse—so you can animate and interact without boilerplate. A public gallery lets creators browse, fork, and remix shaders, turning the site... -
9
notfoundbot
fix & archive outgoing links on your website
notfoundbot is a GitHub Action that helps you automatically maintain the correctness of your website's outgoing links. It finds links that need fixing and opens pull requests that fix them. This action is intended for websites and blogs powered by static site generators. By using post dates derived from filenames, notfoundbot searches for Wayback Machine archives of linked resources that are contemporary to the post itself: broken links in a 2011 blog post will be linked to archives from around... -
AI-based, Comprehensive Service Management for Businesses and IT ProvidersChangeGear provides IT staff with the functions required to manage everything from ticketing to incident, change and asset management and more. ChangeGear includes a virtual agent, self-service portals and AI-based features to support analyst and end user productivity.
-
10
crwlr
Library for Rapid (Web) Crawler and Scraper Development
This library provides kind of a framework and a lot of ready-to-use, so-called steps, that you can use as building blocks, to build your own crawlers and scrapers with. Before diving into the library, let's have a look at the terms crawling and scraping. For most real-world use cases, those two things go hand in hand, which is why this library helps with and combines both. A (web) crawler is a program that (down)loads documents and follows the links in it to load them as well. A crawler could... -
11
jekyll-theme-8bit
A Jekyll theme inspired by classic 8bit games
A Jekyll theme inspired by classic 8-bit games. Jekyll-theme-8bit has been scaffolded by the Jekyll new-theme command and therefore has all the necessary files and directories to have a new Jekyll site up and running with zero-configuration. Snippets of code within the _includes directory that can be inserted in multiple layouts (and another include-file as well) within the same theme-gem. -
12
Nextra
Simple, powerful and flexible site generation framework
Simple, powerful, and flexible site generation framework with everything you love from Next.js. Nextra automatically converts Markdown links and images to use Next.js Link and Next.js Image when possible. No slow navigation or layout shift. -
13
PHPScraper
A universal web-util for PHP
..., including interesting attributes. You can filter and combine these to your needs. In some cases there is an option to get a simple or detailed version. PHPScraper can assist in collecting feeds such as RSS feeds, sitemap.xml-entries and static search indexes. This can be useful when deciding on the next page to crawl or building up a list of pages on a website. -
14
Infosec Reference
An Information Security Reference That Doesn't Suck
... domains, Infosec Reference helps consolidate high-value, battle-tested knowledge into one place, reducing the need to scour scattered blogs or notes. It often contains links to external references, example commands, common workflows, and template policies or checklists. For newer security professionals or teams looking to build a shared reference, it serves as a catalog of contextually verified insights, a starting point for training, or a hub for standardized practices. -
15
LinkChecker
Check links in web documents or full websites
LinkChecker is a free, GPL licensed website validator. LinkChecker checks links in web documents or full websites. It runs on Python 3 systems, requiring Python 3.8 or later. The version in the pip repository may be old, to find out how to get the latest code, plus platform-specific information and other advice see doc/install.txt in the source code archive. If you do not want to install any additional libraries/dependencies you can use the Docker image which is published on GitHub Packages. -
16
Django jazzmin
Jazzy theme for Django
Welcome to Jazzmin, intended as a drop-in app to jazz up your django admin site, with plenty of things you can easily customize, including a built-in UI customizer. 4 different Change form templates (horizontal tabs, vertical tabs, carousel, collapsible). Bootstrap 4 modal (instead of the old popup window, optional). Search bar for any given model admin. Customizable UI (via Live UI changes, or custom CSS/JS). Select2 drop-downs. Bootstrap 4 & AdminLTE UI components. You can add links... -
17
fullPage.js
Create beautiful fullscreen scrolling websites fast and easy
fullPage.js is an easy-to-use library for creating beautiful, fullscreen scrolling websites/ onepage sites/ single page websites complete with all the features you need. With fullPage.js you can add landscape sliders and links to sections of your site, create smaller or bigger sections, use extensions and more! fullPage.js is compatible with all modern browsers and even some old ones like IE9 and Opera 12. It also provides touch support designed for mobile devices and touch screen computers... -
18
DSA Bootcamp Java
This repository consists of the code samples, assignments, and notes
DSA Bootcamp Java is an open source educational repository created by Kunal Kushwaha to teach Data Structures and Algorithms (DSA) using Java. It is designed as a structured bootcamp, covering fundamental concepts to advanced problem-solving techniques. The project provides explanations, exercises, assignments, and practice problems, making it useful for both beginners and intermediate learners who want to strengthen their Java and algorithmic skills. The repository is organized into... -
19
Minimal Mistakes Jekyll theme
Jekyll theme for building a personal site, blog, project documentation
A flexible two-column Jekyll theme. Perfect for building personal sites, blogs, and portfolios. Everything from the menus, sidebars, comments, and more can be configured or set with YAML Front Matter. Built with HTML5 + CSS3. All layouts are fully responsive with helpers to augment your content. Free to use however you want under the MIT License. Clone it, fork it, customize it, etc. Settings that affect your entire site can be changed in Jekyll’s configuration file: _config.yml, found... -
20
TorBot
Dark Web OSINT Tool
... install.sh Now you can run ./install.sh to create the torBot binary. Run ./torBot to execute the program. Crawl custom domains.(Completed). Check if the link is live.(Completed). Built-in Updater.(Completed). TorBot GUI (In progress). Social Media integration.(not Started). -
21
Hugo PaperMod
A fast, clean, responsive Hugo theme
PaperMod is a fast, minimal-yet-featureful theme for the Hugo static site generator, aimed at blogs, documentation sites, and personal pages. It focuses on clean typography, responsive layouts, and sensible defaults, while exposing a large set of front-matter and config options. Built-in features include dark/light modes, archive and taxonomy views, reading-time indicators, breadcrumbs, and an optional profile/landing layout. Client-side search, code highlighting, social links, and comment... -
22
Minima
Minima is a one-size-fits-all Jekyll theme for writers
It's Jekyll's default (and first) theme. It's what you get when you run jekyll new. Minima has been scaffolded by the jekyll new-theme command and therefore has all the necessary files and directories to have a new Jekyll site up and running with zero-configuration. From Minima v3 onwards, the base layout is named base.html instead of default.html to avoid confusing new users into assuming that name holds a special status. Users migrating from older versions with customized _layouts... -
23
YYeTsBot
Renren Film and Television bot, fully connected to Renren resources
Renren Film and Television bot, fully connected to all Renren Film and television resources without deletion of resources. The database of this site is permanently open source and free. You can directly send the name of the episode you want to watch, and you can choose to share the webpage or link (ed2k and magnet links). When searching for resources, it will be searched according to my predetermined priority (everyone video offline, subtitle man), of course, you can also use commands to force... -
24
Polaris React
Shopify’s admin product component library
Our design system helps us work together to build a great experience for all of Shopify’s merchants. Downloads, links, and third-party tools to help you and your teams learn, draft, design, and build products for Shopify. Use the API to build Figma plugins, text editor extensions, or even a completely new site. While we do offer a CSS-only version, we strongly recommend using the React versions of our components. It’s the version that we use at Shopify. It allows for rich, complex components... -
25
WFDownloader App
Free batch downloader for image, wallpaper, video, audio, document,
Use as an image gallery, wallpaper, audio/music, video, document, and other media bulk downloader from supported websites. Also use to download sequential website urls that have a certain pattern (e.g. image01.png to image100.png). Also use app's built-in site crawler for advanced link search or extraction. There is also special support for forum media and open directory downloading. It's a programmable downloader and also works with password protected sites. Say goodbye to downloading one...">